Diary - 2023 in Review
01 Jan 2024My second year-in-review, following the one for 2022!! To write the 2023 review, I revisited my 2022 review after a long while. Some of the goals I had set were achieved, and some I had wandered far away from. Honestly, before re-reading it I couldn’t remember much… and I get the feeling that reviewing on a one-year cycle is a bit too long. It might not be a bad idea to take time to look back on myself every six months.
Actually, after writing the 2022 review last year, I thought it was a great chance to understand a lot about myself. So I started writing weekly reviews — I kept at it diligently until around August, after which they just stopped abruptly (with the excuse that life got the better of me…). Still, a fair amount of a year’s worth of records remained, which I’m glad about. In 2024, I should make it a habit to jot down weekly reviews whenever I can. In this 2023 review, I want to organize those weekly reviews and set goals for 2024. This time I wrote about all sorts of things at length!! Be warned, it’s a wall of text~~
2023 in Review
Getting my first job
Before I joined, I was clearly a Robotics & Deep learning engineer, but now I think I should be called an LLM Developer. Me, doing LLMs…!! Even in the lab I didn’t really know much about LLM research and didn’t pay close attention to it, so it felt strange that things turned out this way. Of course, right before I joined ChatGPT had been commercialized, and after using it a few times I got a strong feeling that the trend was heading in this direction, which is part of why I switched over.
When it came time to be assigned to a department, I found myself wanting to be placed somewhere where I could keep studying reinforcement learning while also doing LLM work. That turned out to be the LLM finetuning team, which I’d heard had few members and an extremely demanding workload. So the moment I walked into my placement interview with HR, I shouted, “Please send me to the most hardcore team!!!” lol. Thanks to that, I was placed well, on the team I wanted. Later, when I heard the story from my team lead, it turned out the lead had to choose one of the new hires, and the HR person came to the lead and said, “I’d recommend this person (me)” (all according to plan…!!).
Over the roughly eight months after joining the team, I learned a lot about LLMs. Before joining, I knew nothing beyond the fact that LLMs are built by stacking Transformers, but now I think I can at least hold a conversation about them anywhere. In particular, since LLM models are so large, there are many methodologies for utilizing multi-GPU setups and optimizing inference speed, and I was able to learn a lot about them. The experience of training using 16 nodes, 32 nodes of GPUs would have been absolutely impossible in the lab (stuff like this is where our company is the best!!!). It wasn’t work that used reinforcement learning directly, but I have hope that I might get to do related work starting next year.
And LLMs really were Data-centric AI itself. Work that touches the architecture and the model, like RoPE or LoRA, was of course important, but the quality of the Data determines just so much. (Personally) I think the biggest factor behind Google still not catching up to OpenAI is MS’s acquisition of GitHub. Also, compared to how I just used open-source datasets in the lab, the large-scale data-creation processes happening in industry were quite grueling work. Especially while I was continuously inspecting data while staring at Excel spreadsheets, I sometimes wondered whether I was really a developer.
In conclusion, I was happy that even after joining a big company, I could do trendy work without much of a gap from the work I wanted to do. Of course, I’m a person with a fairly low threshold for happiness, but lol. And I think I was able to understand at least a little about the “institutionalized systems” that I had set as a goal to experience in 2023 in my 2022 review.
Making plans: a continuous chain of responsibility and communication
A year ago, when I took my first step into society, a lab senior I always trusted and relied on told me something.
In the lab, momentum and processing things quickly were the more prioritized values, whereas in society, handling things conservatively, satisfying each step one by one, seems to be the more prioritized value. That’s the dominant strategy for the majority.
In reality, everyone on my team who I feel is good at their job works this way. They make a plan, execute that plan step by step, and share the results. In my case, that was so hard. Around July, when I was first assigned work on the team, I started working on whatever came my way. It was work related to RAG (Retrieval-Augmented Generation), and rather than approaching it methodically, my thinking was that it would be good to apply and evaluate the latest methodologies one by one and get a feel for it. Honestly, I think the work assigned to me was also a bit naive — not “deliver such-and-such a service” but more like “how about trying to boost the related performance?”
After about two months of repeating these experiments, I felt like I’d lost my sense of direction. I’d gotten a bit of a grasp of RAG, but I’d lost my sense of purpose about exactly what I should be doing… That’s when my team lead said this.
Are you going to drift along vaguely (?!?!) like this until the end of the year? Shouldn’t you figure out what the development scope is, and through what plan and what experiments you’ll validate it, by making a plan and approaching it in advance?
Honestly I’d been feeling it too, but actually hearing it said didn’t feel great. (The buzzword of my year: “vaguely”…) To be honest, the thought “How am I supposed to make a plan when no clear goal has been assigned to me?” crossed my mind, and I felt a little resentful. But even when I tried to make a plan, I didn’t have a good sense of how to do it, and when I asked colleagues around me, the answers I mostly got back were things like, “Why don’t you think about what isn’t working right now, and what you want to make work?”
In the middle of all this, in mid-November a teammate left the company, so I took over their work. It was work that required building a web page, and another team was handling the front-end/back-end development. While collaborating with that team, there was a request to decide on the development direction for a particular feature. If it were a one-off feature, they’d hardcode it, but if it were a feature to be used later as well, it should be properly developed in addition. At that point I answered, “I think I’d need to ask my team lead; I’m not sure myself.” Then the person from the other team said,
Your opinion as the owner matters most. You know the most about it. If you just tell us the direction you’re thinking, we’ll develop it accordingly.
It felt like getting smacked. Right… why can’t I decide this direction? Isn’t it because it wasn’t work I was taking responsibility for? Direction is, after all, a plan. Ultimately, to make a plan, I have to know exactly what it is I’m trying to do. Concretizing the goals a leader presents is the practitioner’s job, and without concretizing them, you can’t make a plan. It occurred to me that the reason I couldn’t make plans was that I hadn’t properly understood the work I was supposed to do and hadn’t taken ownership of it. The RAG work was the same. For the work I was responsible for, I was the person who had to set the goals proactively. Just because I’d joined a company with a hierarchy, was I forgetting that and hoping someone would assign me work, hoping someone would set the goals, trying to ride along on the picture someone else was drawing?
I came to understand why a plan is needed. Making a plan is for sharing my work with colleagues and getting feedback on my work. Especially to convey it to multi-hop colleagues, a plan is absolutely necessary. My team lead has to share what I did at the team-leads’ meeting, and the executives have to pass that up the chain. Didn’t I learn in the lab that even when explaining research to the professor, if you don’t explain it for about two weeks, they won’t understand it? Getting it conveyed across multi-hops takes even longer. Therefore, long-form communication — making long-term plans and sharing results — is necessary. In the end, it was all for the sake of communication.
Taking ownership. Valuing communication. These are the two values I must always keep in my heart in order to drive work in a planned way. Next year, let me try to work a bit more methodically.
How to work with a Strongman
Even as a new hire this year, I got called into a lot of TFs (task forces) convened by executives. My teammates were actually worried I was doing a bothersome chore, but I think it was a really good experience. I could indirectly experience not only what happens within a single team but also what happens at the organizational level, sense how management — not practitioners — thinks, and get a chance to get to know people on other teams a bit better. Above all, I got to experience a lot of how a Strongman (= Decision maker) works.
In TFs run with a Strongman:
- First, the publicly available technologies are laid out bottom-up and quickly organized.
- The Strongman looks at the materials and, exercising insight, forms hypotheses about which direction to go.
- Those hypotheses are rapidly validated by the practitioners. At this point, quantified evidence is needed. For example, if ChatGPT’s function call feature has been released, you must present exactly and precisely, with quantified data, what this feature can and cannot do. Whether the function call feature breaks when there are four or more arguments, whether it breaks when five or more functions are included — if you don’t present quantified data, it’s meaningless.
- This process is repeated.
The decisions the Strongman made in this process were interesting. To leave one example: in a TF, this process was carried out regarding the introduction of ChatGPT function calls. Honestly, I found function calls really fun to study, and I thought it would be good to actually apply them and propose a service. But the Strongman judged that the current methodologies were unfeasible for actually serving a model and generating profit. He judged that while the number of tokens grows exponentially, the performance doesn’t scale proportionally, and that LLMs’ reasoning performance still falls short of that. Even so, he judged that ChatGPT opening up the function call feature was not for the sake of a service but for the sake of securing data. I had simply thought the technologies laid out looked good, but I could sense his capacity to make complex judgments about whether something could be profitable as a long-term service, and what side effects the technology might have.
In the end, to become a Strongman someday, working hard in the short term is important too, but I came to think that thinking long-term a lot is more important. Whether GPT-4’s overwhelming performance is due to RLHF or to MoE; whether, if you apply RLHF so that the LLM follows universal values, it ends up bad at solving heavy-tail tasks; whether we can find that heavy tail — I should keep pondering things like this even in everyday life and adopt a stance of thinking about the future of LLMs. Of course, even if it’s not LLMs, let me live with longer-term thinking about the fields I’m interested in!!
On human relationships
One of the best things after joining the company was that there were many people among my cohort who were similar to me. It’s hard to define exactly (actually, defining such things isn’t good. The truly important things are hard to put into words), but there were so many people of the disposition I like — people who feel resolute and grounded. It feels great that, regardless of business, career, or my own career path, I’ve come to meet many new friends with whom I can simply build camaraderie. Honestly, I think I met a lot of friends like this in undergrad, but after entering grad school there weren’t many chances. Cohort culture is hard to experience when you join as an experienced hire or join a startup, so I think I’m really lucky to have many cohort peers. With teammates there’s a kind of trade-off — getting too close might make work uncomfortable — but my cohort peers were unconditionally helpful. Over the year I received a lot of help from my cohort and got energy from them. I’m so grateful.
In July, I had a good opportunity at the company to attend a lecture by Professor Andrew Ng. The deep learning talk was very general, so I didn’t get much inspiration from it, but I gained good insight about human relationships. It was fascinating that a characteristic of Silicon Valley is “community and together: benefiting everyone, and seizing chances while helping each other.” Shohei Ohtani, who just landed the largest contract in MLB history (and he’s a real cheat character… on top of being handsome), also said, “If you make an effort to become a good person — picking up trash, picking up cigarette butts, taking off your cap and greeting people on the field — won’t a bit more luck follow you?” I shouldn’t forget that living kindly and generously toward the people around me and the many people I brush past in my career is, in the end, a path that lets me become happy.
This year too, I received many birthday wishes, graduation congratulations, and wedding invitations. How fortunate it is to have friends who reach out first when something good happens? It makes me think once again that I should become a person who can congratulate and love someone first, too. I want to tell my precious people — my family, my girlfriend, my cohort, my friends — thank you for making 2023 happy.
How will you live, my friend?
My life
Honestly, in the second half of this year I couldn’t focus on anything much. It’s an excuse if you want to call it one, but there were about two reasons.
First, on May 29th I injured the sole of my foot. A partial tear of the plantar fascia. There’s an old saying that the feet are the root of all illness, and honestly the elders are never wrong. Once you injure the sole of your foot, your quality of life just hits rock bottom. Forget not being able to exercise — you can’t even walk properly, so everything becomes difficult. If a team lunch is set somewhere a bit far, I get there drenched in sweat, and the commute is so grueling… on top of that, the plantar fascia doesn’t even heal. The professor at Samsung Hospital said there’s nothing to do but stay still, but how am I supposed to keep my feet still, Professor?? Do I lie down all day?? Ugh. It was honestly harder than when I had tuberculosis in 2019. Couldn’t play basketball, couldn’t play tennis… and even now in December it hasn’t fully healed. My body is no longer in its early twenties, so when it hurts, let me rest instead of exercising.
The second was the housing issue. If it had been in front of Seoul National University, I’d know the market prices and had looked around a lot, so I’d have a rough sense of where’s good, but since the location wasn’t precisely fixed, I had to scout a very wide area. And since I was trying to move into a somewhat bigger place, there was a mountain of things to study — jeonse loans and whatnot… In the end I came to Wangsimni, which has both Line 2 and the Bundang Line, and I safely wrapped up everything including guarantee insurance for the jeonse loan. I learned the Wangsimni market prices inside out lol. This time I completed the jeonse contract without any help from my parents at all, so I was able to learn a lot about real-estate-related laws and systems. It was quite a tough journey. I have no idea what I’ll do when I eventually buy a place…
Still, I got to live in a two-bedroom for the first time, and my quality of life went up a lot. I’m going out to the company tennis club’s biweekly Friday happy-hours, and somehow my skills have improved a lot, which feels great. Maybe I actually have more talent for tennis than for basketball?? Definitely, without body contact it’s easier. (I’ve been afraid of body contact since the old days…) I also joined a reading study group at the company, which forced me to read about one book a month. Reading notes
My body has definitely become one that can’t exercise as often as when I was at school. Not only am I full of the desire to just rest after getting home, but there also aren’t many nearby facilities where I can play basketball or tennis. So I’m thinking I should scout out hobbies I can do at home. Not just wasting time on Instagram or YouTube, but things I can approach productively. Cooking, for instance.
My career
If I had to introduce my career somewhere, how could I do it? Honestly, my career doesn’t seem consistent (maybe that’s true for anyone). The high schooler who loved physics so much he wanted to go into the physics department ended up going into mechanical engineering to match his grades, then after being discharged from the military, saw AlphaGo and studied deep learning, then went to an electrical engineering autonomous-driving lab, then suddenly went to the US saying he’d do reinforcement learning, then quit the PhD, got a job, and is now doing LLM development. It feels far from a career you can explain with a single thread. On top of that, maybe it’s because I’m doing LLM work at a telecom company, but I think it’s a bit awkward to even explain my current career. Even now it’s clear I’m accumulating “experience” training language models, but can I say I’m an “expert” on language models? What exactly am I an “expert” in?
During the year I also did a few toy projects. I participated in a Kaggle-style in-house data competition, planned an AI-powered calling service and built a demo, and pondered how AI might be applied at my girlfriend’s company. But these toy projects always stalled before crossing a certain level. It made me wonder again, am I lacking expertise…??
The funny thing is that I had the exact same worry in 2020 when I first entered the lab, and in 2022 when I graduated from it. (This is why you should keep a review journal lol — humans are just creatures of forgetting.)
- From the 2022 review -
As the time to get a job drew near, I started thinking about my career. What kind of job will I have, what kind of work will I do, what field will I become an expert in? The final conclusion I reached was the answer I don’t know. It’s funny, but this is the single most important word describing my life.
I don’t know. I have no idea about my future, and I can’t plan my future. Because of that, can’t I live by choosing fun and enjoyable work? Fortunately, I’m fairly quick-witted, sharp, and highly adaptable. I intend to live doing the work I want to do and the new things that come my way. I did research for over two years and lived in grad school. I want to go somewhere new. So I decided to go to the institutionalized big company on the opposite extreme. I plan to stay here at least two or three years. After that, if something I want to do and that looks fun comes along, I might leave without hesitation.
Around August, at the DEEPEST summer camp, I met an ML leader from Hyperconnect. Among the things he said was, “Beware the cargo cult.” Simply going through the steps of ML is meaningless; you have to look at the data yourself, confirm that ML really works, and solve the problems that arise within it. I might move to another company in the future, or do a PhD, but the fact that I’ll keep building my career in the ML field probably won’t change.
When you don’t know what to do, do what you can do right now.
So let me focus on what I can do right now!! Going beyond merely running ML code found on Google, if I keep looking at real data and solving those problems with ML, someday I’ll be able to become an ML expert. Let me meet many people and help them without expecting anything in return. Let me meet many people in the ML field and people who want ML knowledge, and let them make use of me. Rather than trying to make money and gain right away, let me look for opportunities where I can grow even if I’m not compensated!! Isn’t knowing what people need me for, and in what field, the path to finding my expertise?
What do I live for?
Sometimes my girlfriend and I talk about how AI could be applied to her business. Whenever we do, she gets completely absorbed in the story of her own business and the startup scene and pours out the talk. Being so immersed in something is truly attractive. Could I also immerse myself in something like crazy? Could I become the kind of person who can excitedly pour out my own story?
Honestly, I’ve lived closer to a European mindset than an American one. What’s a European mindset? It’s placing weight on life itself rather than on one’s job. It comes out perfectly in the 2023 NBA Finals MVP Jokić’s interview.
“Basketball is not the most important thing in my life. It’s just something I’m good at. To me, getting back home quickly and watching horse racing with my family is more important.”
Europeans are pessimistic about the future. Therefore they regard enjoying the present as more important. I’ve lived with this mindset ever since my exchange-student days in Switzerland. Living fiercely is important too, but enjoying the present is more important. I wanted to live a life with the leisure to take a nap in a field on a nice day, and the leisure to drop everything on the weekend and go traveling even when busy. If you can’t have the leisure to look at the sky while walking down the street, you can’t realize how beautiful the sky is. But that doesn’t mean I wanted to live a life of just getting by day to day. I do have the thought of wanting to be a directed person who lives for some goal, but I don’t really have a clear sense of exactly which goal I should chase.
The fact that I feel like I’ve lost my goal probably has a lot to do with feeling that I’m doing mechanical coding. I’m aware that the work I do helps improve our company’s language model’s performance, and I know I have to keep working to raise the model’s performance. But the question of what this language model is ultimately being developed for hasn’t been resolved. What meaning does contributing to a service or product I don’t use have for me?
Even my girlfriend, the person I know who’s living the coolest life, keeps pondering it: “Why do I live?” Let me keep pondering it. I get the feeling that lately I’m living too thoughtlessly. (My girlfriend tells me there have to be periods like this too… thank you.) Even if not as much as in grad school, let me make an effort to show, little by little, a version of myself that’s developing!!
In 2024…
As for whether I achieved the things I wanted to achieve at the start of 2023, I think I achieved about half. As I’d hoped, I met many good friends, my tennis improved a lot, and within the company I experienced team-level real-world work and even experienced a TF where I communicated directly with executives. I played hard and also pulled weekend work shifts. I was a bit neglectful about reading papers and could hardly do any outside activities, so you could say I was neglectful about keeping up with academia’s deep learning trends.
In 2024, I want to find out what my dream is. A little while ago, while talking with my girlfriend about careers, she suddenly asked me what my dream was. I don’t remember exactly what I answered — I think I said something like I wanted to become a jack-of-all-trades, just answered vaguely. It wasn’t that I wanted to answer vaguely; honestly, nothing came to mind. What is my dream?
A dream, she said, is something you really want to solve — and among those, something that can’t easily be solved. For example, she said her dream is to build a system where female entrepreneurs can get equal opportunities. A hard problem that can’t be solved within a year or two — have I ever had a problem like that? This year, my goal is to live while thinking about what my dream is.
Of course, beyond this, let me live enjoying each and every day. Let me get a bartending license, do the English study I’d put off, and read books. All my close friends are getting married, so let me prepare a bit too.
Above all, in 2024, let me absolutely not get sick!