How to Develop Software: From Idea to Product

Every piece of software—whether it's a website, mobile app, or API—begins with the goal of solving a problem.

As an analogy, let's look at some well-known companies. Facebook was created to solve the problem of social networking and connecting people. Netflix provides a platform that solves the problem of accessing movies and series from anywhere at any time.

I'm writing this article to give a high overview to new developers who wants to start a startup or a project that could become a potential product.

We'll explore the software development process from the ground up.

Research

Research is a crucial first step in any software development process.

This includes indentifying the audience, if there is a market for the product, if the idea is viable, and whether there are enough resources to make it a reality.

This phase answers questions like:

• Is the product necessary?
• Who are the potential users?
• What problems does it solve?
• What are the existing solutions or competitors in the market?

Again, it's essential to understand the problem you're trying to solve and how others are addressing it.

System Design

Before starting any coding, it's essential to create a high-level diagram of the application's architecture, known as system design.

An well-done system design gives the developers a blueprint for how the application will function and scale. It's also useful for onboarding new developers into the project.

1. Functional and Non-functional Requirements

• Functional requirements describe what the app should do. These are major features that users interact with directly, such as user authentication, payment processing, or other business logic.

• Non-functional requirements refer to minor features that the user may not be directly aware of, but are critical for the application. These include factors like performance, security, and scalability.

2. Core Entities

It represent the main objects or data structures that the app will manage, and these usually form the foundation of your database schema.

For example, in an e-commerce app, the core entities might include:

• Users: Information about customers or admins.
• Products: Details about the items being sold.
• Orders: Records of customer purchases.
• Payments: Information on transactions and payment methods.

3. API Design

Determining how different parts of the system will communicate with each other, and how external clients (like mobile apps or web browsers) will interact with the server.

These include but not limited to:

• Endpoints: Define clear, RESTful or GraphQL endpoints for accessing data, such as /api/users, /api/orders, etc.

• Security: Implement proper authentication (e.g., OAuth, JWT) and authorization to protect sensitive data.

• Implement major features

• DTOs for API repsonses

4. High-Level Design

In the high-level design phase, you map out the overall architecture of the system. This includes decisions about:

• Database selection: Choosing between SQL (relational) and NoSQL (non-relational) databases based on the type of data and the app’s needs. SQL is often used for structured data, while NoSQL suits unstructured or highly flexible data models.

• Hosting environment: Deciding between traditional servers (which you manage) and serverless platforms (where the cloud provider manages infrastructure). Architecture design: Whether to use a monolithic or microservices architecture. A monolithic architecture combines all services in one codebase, while a microservices architecture splits them into independent, smaller services that can scale individually.

Minimum Viable Product (MVP)

Teams often come up with many great ideas during the system design phase. However, it's rarely feasible to implement all of them right away. Imagine, trying to implement hundreds of features can take years and without resources, it could be a death road.

What's the solution? Focusing on building a Minimum Viable Product (MVP). An MVP includes only the core features that provide the most value to users, allowing the product to be released quickly. Once the MVP is live and generating revenue, more features can be added over time.

The MVP is key because it allows developers to get real user feedback and make improvements iteratively, rather than spending years perfecting a product that may not meet market needs.

Team and Roles

Once the product's goals are established, the next step is assembling a team. An ideal small startup team might include:

• 4 Developers: Either full-stack developers or a mix of frontend and backend specialists, depending on the project's needs.

• 1 UI Designer: Responsible for creating the user interface and ensuring a good user experience.

• 1 DevOps Engineer: Manages cloud services, configures servers, and handles Continuous Integration (CI) and Continuous Delivery (CD) pipelines.

• 1 Agile/Scrum Master: Ensures that the Agile methodology is followed, organizing sprints (2-3 week development cycles) and facilitating communication between team members.

• 1 Quality Assurance (QA) Engineer: Ensures the product works as expected by conducting tests and identifying bugs.

• 1 Product Manager (PM): The PM communicates with users, gathers feedback, and prioritizes features that deliver the most value. They manage the product backlog—a list of tasks that the development team needs to complete.

• CTO (Chief Technology Officer): Responsible for the overall technology strategy and making high-level tech decisions.

• CEO (Chief Executive Officer): The CEO drives the company's vision, direction, and strategy.

Developers don't communicate with users directly. Instead, the Product Manager (PM) is responsible for gathering user feedback and translating it to tasks (or backlog) for the development team, ensuring the product evolves based on user needs.

The backlog is a prioritized list of tasks and features that the development team will work on, with a focus on delivering value to users and generating revenue. The Product Manager continuously updates the backlog as new feedback is gathered and priorities shift.

To manage the development process, the Scrum Master (or Agile lead) organizes the team's work into sprints—time-boxed periods, typically lasting 2 to 3 weeks, or depending on the team.

CI/CD pipelines

If you'd like to take a look CI/CD pipelines as code, article coming soon!

Continuous Integration (CI) is a practice where developers regularly integrate their code changes into a shared repository. In a team, it really helps maintain code quality, speeds up development, and reduces the risks of deploying unstable or buggy software.

Typically, the DevOps engineer sets up the CI workflow with Github Actions, or Jenkins, or other similar service. But also a single developer could set this up.

There are many ways to set up CI pipelines, which depends on the team size and organization.

Truncate-Based Approach

This approach works very well with small teams.

In TBA, there is a single main (or trunk) branch where all changes are continuously integrated.

Continuous Integration (CI)

In this approach, the developer needs to checkout off main to a feature branch. Then, merging with an PR to main branch when done and getting approved by another developer. This process repeats very oftenly (maybe 10-20 times a day).

To avoid bugs while constantly merging to main, it's important to test before merging. Github actions would do the work, which triggers an event when PR is created. These events start some processes such as testing, build, deploying, etc, in the background. If the process fail, it prevents merging to main.

The advantage, this allows the team to release features quickly in response to feedback. Even if bugs are found, a hotfix PR can be merged rapidly.

The goal is to minimize the time to deployment.

Continuous Deployment (CD)

Now, let's explore Continuous Deployment (CD) with this approach in more detail.

Not all CD strategies deploy immediately after a successful merge into the main branch. Instead, some rely on tag releases to trigger a deployment.

Tag releases allow us to track stable versions of the application in production. When a new tag is created, it triggers a GitHub Action that starts the deployment process.

One advantage of this approach is avoiding the introduction of bugs into the production environment. The pre-production environment allows QA, stakeholders, and test users to catch bugs before it's deployed to production. It's always better to catch issues in pre-production—your PM will thank you!

What about drawbacks? One could be merging unfinished features. Some features take longer to complete, but with this approach, feature branches must live short and merge oftenly.

The solution? Feature flags. These allow you to safely merge unfinished features by keeping them disabled until they're ready.

Gitflow Workflow Approach

This approach is recommended for enterprise or larger teams.

Now, there are three distinct branches (dev, test, prod.), each serving a crucial role in continuous integration. Each branch operates at a layer below the next, with the top branch being production, where the final product is deployed.

Take a look to the workflow:

1. Task Creation and Checkout:

   • A developer picks up a task (e.g., Feature A) and checks out a new feature branch from the dev branch.
   • The developer works on the feature independently, committing code changes to their feature branch.

2. Pull Request to Dev Branch:

   • Once the feature is complete, the developer creates a pull request (PR) to merge the feature branch back into the dev branch.
   • The PR triggers a series of automated steps using GitHub Actions, which initiate build, test, and deploy processes to ensure that the code is stable.
     • The tests run in a dedicated dev environment that uses fake datasets for testing purposes. This allows the developer to quickly verify that the feature works without using production data.
   • If the tests pass, the feature is reviewed and merged into the dev branch.

3. Staging Environment:

   • After the feature is merged into the dev branch, it undergoes further testing in the test/staging environment. This environment is set up to closely mimic the production environment, but uses realistic datasets for more accurate testing.
   • The PR triggers additional tests, such as:
     • Load tests: Here we measure how well the feature performs under stress (e.g a lot of computational calls).
     • Smoke tests: To quickly verify that the basic functionality of the system is working.
     • End-to-end (E2E) tests: We ensure that the feature works as expected from the user's perspective across the entire application.
   • If any of these tests fail, the process stops, and the issue is reported back to the developer for fixing, otherwise, the feature is ready for step 4.

4. Pull Request to Production:

   • Once the feature passes all the tests in staging, a new pull request is created to merge the code from the test/staging branch into the production branch.
   • This again triggers GitHub Actions, which run a final set of build, test, and deploy steps to ensure that the feature can be safely deployed to the production environment.
   • The feature is then deployed to the production environment, making it live for end users.

5. Production Environment:

   • The production environment is where the final, user-facing version of the application runs. After successful deployment, the feature is available to all users.