Concerns about SW Architecture in a Startup

(This article is a re-post from my post in July 2016.)

Working alone as a full-stack developer in a business, my biggest concern was, “What am I actually developing?” Without an overall architecture, I blindly trusted a full-stack framework and excitedly developed, but when I looked back, I hadn’t made any progress. It’s strange, a prototype should have been out by the end of last year, but it’s already been delayed for half a year. Of course, unexpected paperwork while running the company was the main cause, but I think the bigger reason was that I spent too much time trying to solve unexpected issues that arose during development.

A long time ago, in the SI era, I could be put into practice and develop after only reading documents written by PMs or PLs, or after a week of training at most. Even if it was beyond my understanding, in the days of Spring 2.5, I could add a new service by simply copying the MVC structure developed by a senior and changing a few forms and models in the UI. In other words, it was copy-and-paste. We would open it like that, and if the data got messed up later, we would hurriedly fix the service overnight. The important thing is that the basic design was well done, so adding a service was so easy (even if it wasn’t perfect).

But the situation has changed. To be honest, the two or three projects I worked on as a freelancer were all delayed. Of course, I want to call the reason unexpected exceptions, but looking back now, I feel that a lot of it came from a slightly ignorant perfectionism. The service hasn’t even opened yet, but I’m already worried about MITM (Man-In-The-Middle) attacks, SSL hasn’t been adapted yet so I’m worried about encryption, MySQL’s password function has been hacked so I’m worried about password encryption… Not only security issues, but sometimes I wonder why I have this aggregation/composition, why normalization isn’t done properly on this DB schema, why unnecessary N:N relationships have been added… While fixing these parts, not only did I exceed the expected deadline, but one project was even in danger of being canceled because the development wasn’t done on time. It’s all because of my ‘greed’ for development.

Now I’m even doing a startup and developing a product alone. I was weak in front-end technologies like AngularJS, so the learning curve was steep and it took a lot of time, and while developing the back-end and understanding the RestAPI structure, I had to understand and apply CORS/CSRF/Token. For some reason, Scala’s Future or Promise seemed suitable for the web structure, so I used Slick and made everything asynchronous. Of course, in this process, it took time because I wasn’t familiar with Scala, and it took time because Slick’s JPA-like model wasn’t well understood… Fortunately, it wasn’t delayed that much because it was the beginning of the business, but I thought that if this continued, development wouldn’t happen at all.

Why did this happen? First of all, I blindly trusted the full-stack framework. Open source was overflowing, so I thought it would be well managed and patched it here and there. Then dependency management didn’t work, so I learned Bower, gradle, sbt, and developed again. Still, there was no unified standard. In the end, I needed to study. No matter how many years I’ve been developing, asynchronous on the server-side, not in Ajax, or the queue method is not well understood.

The most important thing is that there was no architecture document. Last time, while reading Jo Dae-hyeop’s book ‘Large-scale Architecture and Performance Tuning’, I found it strange that I was developing without such a design document. Do you want to develop or not? What’s the use of setting up a development environment and a CI/CD environment without even a common document? At best, I had an Evernote page with IPs and accounts organized, ERD, UI prototypes, and a business plan. It’s amazing that I’ve been able to develop to some extent in this situation.

So, I really felt that if I continued to develop like this, the project would go astray without even a prototype coming out. So I decided to re-read the book and design the architecture, and after about 10 days of struggling since last week, I was able to create something to some extent.

The Beginning of Architecture Design

The beginning was not smooth. First of all, according to the book, Business Architecture and System Architecture were needed. Except for ERD, I had no System Architecture at all, and Business Architecture was too broad to be replaced with a Business Plan, as is the case with most business plans. So in the end, I had to create a new one.

It’s hard to disclose because the business is in progress, but the rough flow is 1) Summary 2) Product 3) Market Analysis 4) Core Functions 5) Domain Model 6) Overall Architecture 7) Milestones 8) Team. In particular, for Core Functions, I rewrote it in User Story format, As a <type of user>, I want <some goal> so that <some reason>, to make it easy to transfer to JIRA. In this process, the core components were divided, and the overall relationship between each component was visualized and expressed again. Then, up to prototype/version 1/version 2/version 3, we divided the importance with a period of about 5 years and wrote a plan for this.

In particular, I thought about the organizational structure in this process. At first, I didn’t even think about expanding the team, but I’ve been feeling lately that I can’t do the project alone. Then, if I have to expand the organization someday, what kind of team will I have..? When I thought about it, I’m not interested in titles or anything like that, and I want the overall relationship to revolve around my own responsibilities.

After checking various corporate cultures, Google’s one felt attractive to me. Responsibility transfer per product unit, in fact, in the organization chart I designed, the design team supports the detailed tasks of other teams, but in fact, this is not exactly correct. However, I want the design team to be a little more systematized and a team that can create independent UI components. I thought it would be good to unify the overall design standards, so I took it out separately.

But the important thing is that team building is not really necessary now, and at least for this year. The important thing is which components will be the core and what parts are really beyond my (the author’s) capabilities and require experts in that area. It was a great help in understanding. So, I felt that teams related to design, architecture, and security, which I felt were lacking, were absolutely necessary. That way, I felt relieved that I wouldn’t have to struggle while developing the prototype at least now.

System Architecture Design

After finishing the business architecture for about 4 days, I started designing the system. I thought it would be done quickly because it was just a matter of diagramming what I was already developing, but it took almost a week. The overall table of contents is as follows.

1. APPLICATION ARCHITECTURE
   • STATIC ARCHITECTURE
   • DATA & ACTION FLOWS
   • DETAIL ARCHITECTURE
   • PROTOCOL DEFINITION
   • MESSAGE EXCHANGE PATTERN
   • REST API DESIGN
2. TECHNICAL ARCHITECTURE
   • DEVELOPMENT ENVIRONMENT
   • BIG DATA PROCESSING
   • DEPLOYMENT ARCHITECTURE
   • NETWORK ARCHITECTURE
   • FRAMEWORK ARCHITECTURE
3. DATA ARCHITECTURE
   • DOMAIN MODEL
   • ENTITY-RELATIONSHIP MODEL
   • NOSQL DESIGN

Since everything followed the book, the table of contents was easy to organize. However, I needed to prune out unnecessary things right away. For example, using Google Cloud, I didn’t consider server specifications or networks. I started to create it focusing on what I really needed, excluding the real bare-metal server architecture.

The most important thing is to clearly distinguish components, accurately define data, and clearly visualize the relationship between them. There were several times when I was confused by the existing components while developing. As an example, in terms of managing the API Key of the user’s Open API, for SNS Key and Healthcare (Fitbit-like) API Key, although they have been recently unified with OAuth 2.0, they were stored in different DBs. (The former is Account DB, the latter is API DB) As a result, two services and DAOs were created, and even two REST URLs came out, so even the front had to be created two. Later, I understood and hurriedly fixed it, but in order to understand it, I had to draw the entire architecture on paper again or analyze it through forward engineering and fix it.

In fact, that was the biggest problem for me, so it was the core of this design. By dividing the entire components into tiers of about 3 stages and considering the organic flow of data generated here, I started to see what models were needed and how the models should be related to which components. In the end, in this process, the component areas corresponding to MVC were organized, and as the models were organized, the ERD was also organized.

So, these organized documents, although I mainly used Keynote, were captured or otherwise organized in Confluence. I used Redmine before, but Confluence is also fun because it has a lot of plugins. And finally, there was something to show when someone joined the project later. No, at least a web document that I won’t get lost in.

Atlassian Confluence seems to be a great Wiki the more I use it.

Finishing the First Stage of Architecture

So now I’m working on detailed design. For the components that need to come out before the prototype, I’m defining 1) function definition 2) procedure 3) UI/UX Prototype 4) Data Flow 5) Related Data Model for each individual component. However, if I finish this and develop it, the product doesn’t seem to come out at all, so I stopped here for the first time. I think I should do it little by little whenever I have time while developing.

Literally, I finished the first stage of architecture. The question that kept popping into my head was, “Is there a perfect architecture?” And the thought of whether software can come out within that perfect architecture. I also invested 10 days to grasp the big flow and fix the lacking parts, but I felt a little bit like, “What did I do?” because there was no detailed design. Come to think of it, in the projects I’ve participated in, PMs or planners often changed the design and developed it in a somewhat incremental model. In the end, it can’t be perfect, but I keep pursuing perfection.

In the end, there is no answer anywhere. I have to give up the idea that a perfect product will come out. As much as the big function of DevOps is quick feedback, development, and response, shouldn’t the architecture of a startup be faster than existing development? But if you develop alone, you have to do the design if you do the design, and the development if you do the development, but doing both at the same time consumes too much energy and it doesn’t actually work well.

So, you have to cut out what you need to cut out. Even in the current detailed design, I can only roughly create UI/UX prototypes, and I have to continuously share and use large models. Data flow and procedures are the core algorithms of the program, so I have to design them in detail. If this is not done properly, the speed of developing the program itself will be reduced too much. Since there are no UML or Class Diagrams, these OOP problems have to be solved in my head.

STEP-BY-STEP of My Future Development

1. Detailed architecture design
   • Design the corresponding data model as there is an overall flow
   • Procedure (algorithm) design
   • UI/UX Prototype
2. Model development (refer to model design)
   • Create DB schema
   • Design/create model
   • REST API design/production
   • Model development to be used in Frontend
3. Logic development (refer to UI, procedure)
   • Backend Controller/Service/DAO development
   • Frontend View/Controller development
4. Test (I’ll have to develop test cases for this someday…)
   • REST test, binding test, DB information test, etc.

I also thought, “I should create a perfect prototype and start posting,” until I posted it on Brunch, but it took a long time. I have to release it quickly and get QA. So in the end, you have to cut out what you need to cut out. Therefore, startups cannot afford to be perfectionists.

Anyway, I think I’ve learned a lot that I need to study architecture endlessly. I have no choice but to keep improving it in the future, and I think architecture is absolutely necessary if the organization grows later. It seems like it was an adventure in architecture design that made me realize that.