junjizhi / junji-blog

My own blog

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Builder Logs

junjizhi opened this issue · comments

Day 1: Start builder logs

What does SUCCESS look like?

Based on all your experience (don't judge), imagine the kind of success I long for:

  • Demo-ing the software that actually solves the problem for users (simple, and fast). Then I would say: I build it!

I have the logs to tell about my humanity

  • Bring out the best part about me

Less obvious success

  • Built something marvelous in terms of architecture / design
    • Future engineers wow about my code

Why: Logs bring persistence, overcome dev numbness and many other benefits

It's a hack to get myself to write and persist.

The logs here cater to my needs to feel

  • meaning, thus motivated to walk longer
  • nostalgia
  • connected with the best myself
  • seeing each small step made in the past
  • powerful lesson learned & reusable
  • persisted memory
  • verifiable words
  • Able to let go and MAKE PROGRESS

And having Github issues comments, I could start things fresh, and build things on top.

Dev fatigue

Being overwhelmed by work, and feeling less connected with humanity.

All I'm doing is execution, execution, and more execution. Overworking makes me exhausted, and makes me doubt the meaning of my work.

Letting go of cognitive residue

We grown-ups live in a complicated world, get exposed to many aspects of life.

I have some many flashbacks, I have some many cognitive residue that I'm afraid to let go.

Letting them go in the words is the way to solve the complex world problems. I can relax and sleep well at night, knowing that I have put down the important things, instead of needing to keep them in my heads

Ideas come from reading stories about Tony Hsieh

  • He tried many business ideas, and success didn't come overnight
  • No matter how fast grow the company / idea is, we still need the execution on it
  • I don't think opening Pizza shop is something I would do now. 10 years ago? Maybe. But not now.
  • Finding something that are true to me, and bigger than me, that really resonates with my inner thoughts, than that's happiness
  • Some ideas are just simple. Like:
    • making shoe shoppers happy
    • build things that help people see better ads
  • One day I would write about:
    • projects I tried
    • businesses I tried
    • efforts I tried

What: The logs here are about building

The logs would be about:

  • current problems I'm facing
  • what books I'm reading

Get all the logs into one place. It could be a journal, reflection, learning.

How: Write casually, but truthfully

Jot down my thoughts, once at a time.

Day 1: Thinking about Argos

Not so fun when coding alone. I felt less motivated.

Q: Should I bring in another person?

  • I'm afraid to deal with people letting me down in the short term, or people having different opinions
  • But I want to use people's strengths to help me propel the project forward

Who are the people I want to bring on board

  • Ken
  • Victor

How does SUCCESS look like?

  • Demo to clients and we get our first order.
  • Get investors interested, and we have money to hire engineers.
  • Some parts of the system has reusable parts, and build a different business around it!

These logs are closer to humanity, more important than what's built

It is the process of building, overcoming all kinds of hurdles, difficulties. Spinning the problem in a way that makes us sustainable. That's what's the most valuable part of this experience.

What's closer to humanity is generic, forms a dialogue that everyone can take part in.

Then I have something to write, and you have a reason to listen.

Day 2: Too busy with work, and didn't make any progress on building

  • I worked all the way until 8:30pm.

Know when to stop.

What's considered builder progress?

  • Git commits
  • Writing about new learning
  • Writing with new observations

What's my system of building?

  • Thinking

What idea can I borrow from content generation?
For content creation:

  • Set goals
  • Figure out buyer personas (What they want)

Day 3: Github may be the best tool for writing logs like this.

  • reuse the top PR as a checklist
  • append logs there, and make it easier to find.

Day 3: Argos: Building topology control in D3

Ticket: https://github.com/junjizhi/argos/issues/21

Approach: Start from simple, and build from there

Investigations

  • Available open source projects?
  • Challenges of implementing topology control in D3?
    • Who controls the data?
    • When should data selection be communicated back to backend?
  • Network visualization basic features
    • Nodes filter
    • Static network graph, no gravity
  • Network visualization advanced features
    • Refactor to be usable for other data

d3 implementation

Implementing controls

d3.select('#center_XSliderOutput').text(value);
forceProperties.center.x=value; 
updateAll();

Summary:

  • Use d3 selector
  • update global variable related to force setup
  • call global function to update all the things

Thinking: If updateAll() doesn't have any magic, we can wrap it in any event handler, and call d3 to update itself.

Q: How is updateAll() implemented?

  • update simulation object, which is a global var
    • what is simulation object?
  • update display, which is setting node and link variable
    • What are node and link? Why singular?
      • A: svg objects
    • Why separate them? Can we rename them?
      • nope. They are the appended objects after we draw the graphs. We would need to get a handle of them to update the chart.
      • separating them are necessary, because you need link object to update all links once user configuration changes
    • How to render data from d3?
      • svg.data(graph) <= graph comes from d3.json() call
      • d3 data object is init after it reads json file

Q: How to implement node filter UI?
A: A dropdown list that controls the nodes data, and pass it into the d3, and make it redraw.

How to implement static network without animation?

https://stackoverflow.com/a/47522074/1241108
Gist: Stop the simulation right after creating it.

Argos implementation thinking

  • The node filter is a state in VueJS

    • v1: node filter is transient. If you refresh the page, it would be gone
    • v2. node filter is smart and savable. next time if you re-apply the node filter, it smartly checks the nodes in the list, and uncheck the ones not in the list
  • Vue + d3: How does it implement the d3 package?

Experiment

  • Make up the fake graph data
  • Render it in d3
  • Render a node filter widget in vue
  • Pass the node filter state to d3

Render node filter in Vue

  • For bootstrap-vue, we can use checkbox group
    • how do we make a pane with scrollable bar in bootstrap?

EoD update: Implemented node filter with Bootstrap checkboxes and render chart!

Day 4: Interface to sync messages

  • events
  • performance

How to consume messages from Kafka.

incremental updates are more friendly supported.

Day 5: Vue slots

Definition

  • a way to distribute content
  • defining a component
  • When you use the component, you can pass in the content when you call it
  • then in the component definition, you add <slot> </slot> to insert the content there

What is v-slot?

A way for the callsite to pass down named slot

<base-layout>
  <template v-slot:header> => we want this content to the slot named 'header'
    <h1>Here might be a page title</h1>
  </template>
</base-layout>

Day 6: Improved UI for topology network

  • Add keyword filter
  • Select all / only nodes
  • Use network icons

Day 10: True DEEP Dive in vue-d3-network

Questions

  • How is d3-network organized?
  • How is it rerendered?
    • Does it have performance problem?
    • Is it the best practice?
  • What does it mean for our projects?

Extra

  • How is d3 integrated with React / Vue / other frameworks? Any problems? Any caveats?
  • How VueJS plug-ins are published?
  • How to implement controls in d3?
  • What is canvas renderer vs. svg renderer?

Things NOT to do

  • Publish blog articles based on the commits / problems solved from argos

Diving

  • How is d3-network organized?
    • a long vue component that does rendering
      • It doesn't use template, but directly overrides render function to render SVG (or canvas) elements from d3.
    • wraps the d3-forceSimulation in the simulate call.
      • d3 simulation is all we need
        • What does d3 force simulation needs as input to run the simulation?
      • svg-renderer is another class that controls the rendering
        • it holds a bunch of props about the network graph
        • svg render also has a template to render the svg code (do we need to it manually?)
          • That's necessary because they want Vue to control the rendering of svg elements, including binding click listeners, style, classes, etc.
          • How does d3 books generate the initial network?
            • vue-d3-network has two places it builds nodes, so now i don't know the structure of the code any more
            • we pass in a list of nodes as a props
            • from nodes in props, we watch the changes, and buildNodes(), which is simply building the nodes model internally, which would be passed to simulation to build nodes.
      • when DOM is ready, call animate(), which calls simulate()
    • What are the properties / control knobs for d3-forceSimulation in vue-d3-network?
      • What's missing compared with the full power?
    • What triggers the rerendering?
    • How can I make it rendering static?
  • What's the convention for implementing UI controls in d3 simulation?

Learning

  • Use VS Code to open repo is more effective to grok the code
  • Typing down the questions is effective in understanding
  • When asking questions, look at what it does first, and avoids shallow questions.

Day 11: Accurate, functional, clear, distraction-free content is always an asset

Even after ten years, people are still bookmarking my article. History is too short for not making meaningful content.

Making niche content is powerful.

Day 12: Load performance data into UI

  • How to round decimal places with command line?
  • How to adjust the chart width?

Processing data command line

cat performance.txt | awk '{print $1"|"$2"|"int($3)"|"int($4)"|"$5}' FS=\|

How to style vue-echarts?

  • We can adjust the style by changing the .echarts style in the component

vue-echarts working logs

  • Observation: It can handle timestamps as strings

    • How much can it handle?
  • Take the first 1k rows, and show them in the chart

    • Show first 500 rows
    • How to limit the number of rows?
      • Enforce in the UI?
      • No limit?
  • Add chart legends

  • Show how many charts?

    • Dynamically render the lines in one chart
  • Add chart illustration

  • Add chart vertical line

  • Drag to select a timespan and update filter

  • How to load data into UI? Options:

    • JS code to read from csv file
      • Can't do this because web app has cross-origin limitations
    • Convert csv into json object, and load it into memory when file is added
      • JS: How to read object from json?
    • These are stub data any way, so the logic would be replaced

Next steps

  • Show performance for certain node only
  • Correlate the data with nodes from topology tab

topology tab

  • Zoom in/out

Day 13: Continue performance data UI

(Appending notes in previous day)

What can our data analytics UI be used for?

  • Data analytics for other industry

Saw nextcloud, want to give it a try

discussion with wecise folks

  • sliding window
    • down / up for the 24hours, and that counts as an alert

Day 18: Chat with Geoffrey about Argos UI

  • Events table: Use IBM UI
    • refresh
    • node selector
    • view button
    • view dropdown
    • column picker (a popup modal)
    • Rows
      • trim the long lines
      • group events by parent ID
    • Columns
      • adjustable width
  • Event details sidebar
    • alert fields
    • alert details
    • journal

Day 19: What I need from Geoffrey

  • Filter builder
    • Show me the advanced tab, and what filters are there
    • Do we need to implement those
  • Column configuration
    • Where do you configure the columns on the web version?
    • Views, does it include column configuration?
    • Ordering of the columns
  • Relationship between global search with in-table search?
    • view = a MQL query
      • columns
      • where
      • sort
    • When saving a view, we could compose MQL from the UI
    • But we couldn't reverse MQL back to UI configs
      • Solution 1: Store three things separately:
        • keywords / MQL
        • UI views
      • Solution 2: Build AST of MQL, and recover from there
      • Solution 3: Global search & intra-tab search are separate things
        • global search
        • events filter
        • problem: you can do search and filter on both places, which can be confusing to users
  • Default columns

Asks from Geoffrey

  • sidebar to add filter (100+ fields, one search box to search both field name and value)
  • sidebar to add sort
  • sidebar to add journal tab
    • refer by identifier
    • uneditable, imported from dump
  • move alert details (raw) to fields
  • right click to add a few options
  • assume: all events all have severity fields
    • retrospect the field column
  • graph: if selected one node, and show all one / two hops

Need from Geoffrey

  • get a list of hard-coded Netcool fields
  • Geoffrey: think about moving node dropdown to global(?)
  • Discuss how to enable raw advanced sql checkbox, or parse sql

Day 20: How to work with people

Goals are important

  • Deliver good software
  • Encourage with people to work on this piece of software

Working with Rurui

  • Pair programming
    • Show him TDD
    • Show him quality craft
    • Show him the way of thinking
    • Coding styles
      • break long files
      • break long lines into multiple short lines
  • Share with him the deployment

Day 21: IBM NPI Performance & Topology product analysis

  • Each performance is based on the device, interface, and time range selection
  • Select a category aggregation
  • Always show top ones first and allow people to drill down
  • In the performance chart, show the historical data in the bottom chart

The insight: Should we upgrade the device, or how to regulate the QoS of the app for this device.

Overview

  • Congestion
  • QoS
  • Traffic

Summary

  • top 10 interfaces (?)

  • specify a time range

  • congestion section

    • outbound pkt discard deviation
      • total packet drops per queue (by drilling down to an interface)
    • inbound pkt discard deviation
    • top 10 apps by total delays (confusing)
      • have to select an interface
  • QoS section

    • have to choose a source and a destination
    • round trip time deviation
    • probe loss deviation
  • Traffic section

    • outbound ut. deviation %
    • top 10 app by total volume (octets)
    • inbound ut. deviation %
    • top 10 ap by ave. utilization %

Video: Overview to drill down (45.02)

  • click on Top 10 app by total volume chart title -> traffic details page
    • device selection
    • interface selection
    • direction selection
    • time range
    • preselected aggregated metric

App KPI

  • rank
  • app name
  • business relevance
  • app group
  • category
  • sub-category
  • total octets
  • packet#
  • ave utilization %
  • max utilization %
  • ave throughput
  • max throughput

Time filter

  • Last hour
  • last 6h
  • last 12h
  • last 24h
  • last 7 days
  • last 30 days
  • last year
  • custom

For Geoffrey:

  • Confirm the performance drilldown dashboard is the one we want to build
    • Device health?
      • historical data for the last hour / last day / last 7 day at the bottom
  • Term questions:
    • Device is ip or node?
    • Entity: interface.

Not supported features

  • IP grouping based on configuration
  • configure app information (groups, biz relevance)
  • select built-in views of charts (top destination, etc.)

Topology

  • three filters
    • node list
    • hops
    • element to element / host to host / element to host
      • What is element to host?
  • Topology graph interactions
    • right click on a note, and recenter the node
    • reorg the tree hierarchy
    • zoom in/out
    • filter
    • timestamp slider
    • undo
  • Node filter
    • type, and auto complete

Day 22: Imagining the success of our product when we get orders from clients

  • Not that hard to develop UI

Researching on our tech partner

  • founded in 2016
  • Dr. Zhigang Li has 20+ years of dev experience in IBM, but not many papers
  • no papers found (so we can't find many things about him)

Discussion with Tianzheng

  • Easy to do with Graph + relational database?
    • How to implement MQL?
  • Any research value there?

Update: 2020-12-29

Done well:

  • Didn't talk too much, and to the point

Tianzheng

  • They have a prof who did Graph database research before, can ask them
  • not so complex to implement: tweak the parser, and optimize the query planner
  • Sent some papers, worth reading.
    • Academia has some progress on this topic, so it's worth reading

Geoffrey UI update

  • Ultimate: Dashboard builder UI
  • A charting library that knows how to render the MQL results => Google Data Studio
    • top something
    • depends on the shape of the data returning from MQL
      • table data: tables
      • topology
      • performance -> chart
        • with the chart interactions
    • With this, we can build a render engine based on your configs
  • Use bootstrap to build cards UI.
    • Add a new card as a new query (it is based on raw query)
      • or a set of queries?
  • Q: How would they use the graph queries? Is it really that useful?

Geoffrey questions

  • can we link the entity by platform
    • what do you mean by app by connection?
  • how to do fuzzy search in MQL?
    • pre-configured a list of columns?
      • they support one-key search, but up to front end to implement
      • work together

Day 23: What Geoffrey really wants is a rendering engine

Input

  • MQL
    • filter
      • time range
      • relational constraints

Output

  • Events table
  • Performance
  • Topology

Questions

  • URL bookmarkable?
  • Top level saved views (?)
    • presented as tabs?
  • Exportable?
    • How to generate reports from here?
  • Blocks UI
    • based on one MQL
    • similar products
      • Jupyter notebook
  • Raw MQL viewer

Implications

  • Performance: The top ones, they can't be pre-set. Or the UI can only be for demo purposes. The actual UI, we should allow them to enter a MQL with the minimal changes, and they can preview the results.

Discussion with Zhaodong

API

  • What does it return?
  • Does it always return everything? Does it join behind the scene?

Deployment

  • How to deploy to the platform?

一键搜索

  • unknown columns

image

image

Indie hackers: Bundling and Unbundling

When bundling works

  • Big company with lots of existing customers
  • Product are homogeneous, and replaceable
  • People want simplicity, bankrupt by tons of options already

When Unbundling works

  • Companies acquired by big ones, and sentiments worth riding
    • biz model de-risked
    • exodus, worth catching the opp
  • Opportunities
    • B2B: Workflows on Excel
    • Website builders
    • Academia: Unbundling research
      • Math phd: why paying for pens and papers? Lots of money as tuitions

Zhaodong m3/wecise api

generate m3 api

  • INPUT
  • OUTPUT

search

  • class -> optional
  • lucent -> tokenizer, fulltext search
    • no matched columns

data shape

  • data
  • meta
    • classes
    • columns -> has types
    • relations
    • rootclass: top most parent class
    • words: search keywords

dfs

  • write directly to files in the platform

Deployment

  • dfs.write('/path/to')
    • very low level, and we have to establish the convention on using the platform.
  • script/matrix/js -> api
    like an OS
  • too generic
  • omdb -> map (key-values) -> have to be serialized

script api debugging

log.debug
log.info
log.warn
  • es5 js flavor, no const, let.

image

Day 25: Want to go for a PhD in database

That would unlock the potentials of going back to China. Cross an item in my bucket list.

长期来说,学界可以给我的是短期内探索任何方向的自由。虽然现在我也可以读 paper,了解到底是什么。

从问题本质来说,我需要的是

  • 理解最核心的问题,然后再去扩展到具体的 paper

数据库核心问题

  • how to implement query processor / compiler / optimizer?

paper 都是一种 conversation,告诉别人你做了什么

idea: a tool 来问题,然后得到解答。能够做到具体到哪段字句,然后得到答案,就像学术界的 stackoverflow,然后人们可以留言。

搞清楚我的 motivation 再去做这件事。

  • 名誉
  • 智力活动的追求

最怀念的是,对知识学问平等的追求。在一个小领域得到

  • 有益的指导,让我迅速学习
  • 有益的讨论,深入学到知识

学界的优势

  • 安静,孤独的探索

Day 26: Continue building on UIs

  • Add filters: allow users to select Node/interface/metric/time windows
  • Add commonly used: top xxx charts
  • Improve topology chart
  • Add cards UI
  • Deploy argos to m3 platform
  • Wire up APIs
  • Add MQL checkbox

Building performance UI

  • node list is the complete list of nodes?
    • or it is just the list of nodes in the search result?
    • Design:
      • when the search is empty, it pulls the complete list of nodes (capped at 50)
      • when the search is present, it narrows down to the ones in the search results
  • We get the node list from events?
    • They should always select a node to see the drilldown
  • What about overview?
    • We would build the overview later
    • currently it is showing single node performance results
  • Interface is depending on the node dropdown.
    • let's delay the implementation for now
  • so selecting time, and node, then show a graph
  • Where to show metrics?
    • a: a drodown in performance tab

Remaining questions about events UI

  • both places can save user data. Within events tab, there are a bunch of preference data there. they should be saved too?
  • So we add a global node filter. It applies to the global data input into the table
    • events table has its own filter/sorting/fitlering, etc.

Chatting with Geoffrey

Global search

  • Add radiobox to indicate it is full text search or MQL search

Global filter

  • Node
    • will need to have interface
  • App groups
  • Support group
  • Datadog style time range filter
  • Search history tab

Events views

  • Update seeds data to have common set of nodes
  • Apply global node filter to events table
  • Table local filter should apply to global nodes filter (?)
  • Local views (apply to events tab only)

Performance tab

  • Node filter applied to performance tab
  • Update performance seed to have some data
  • Update time filter, it should update the results of performance & events
  • KPI Dropdown
  • If multiple node / interfaces selected, the KPI dropdown is the list of common elements
  • render in one chart
  • more legends to show
  • (future) save button (send to dashboard)

Topology tab

  • node associated events
  • link associated performance

Day 27: Indie hackers, motivation, startup approaches / Inspired by GoRails, content creators

Content creation approaches

Know what it takes first

I can't seem to pull this through. It is a new-year-resolution-ish thinking. I maintain a blog, published some courses, but didn't get a lot of traction.

Find my own rhythm: Force myself to record daily

Win trust

If I want to sell content, I need to win trusts among people first. Produce something:

  • better organized
  • explore deeper
  • useful

Find sth new and keep publishing weekly: Weekly screencast

What's working now: Building something. I felt joy of imagining users using it

Leading a team

想要成功,就要尽快发现用户需求,然后找到合适的产品来满足,最后可以自己为自己打工

Geoffrey new UI designs

Tasks

  • - Little icons representing queries (?)
    • don't need it for now
  • Move time selectors up to the same line as global search
    • Global time range picker TBD
  • Global filters: Change it to icons (TBD)
  • Search options tool bar
    • radio box
  • Events tab
    • quick stats view
    • modal mockup
      • timeline view (?)
      • Grouping of events
  • Performance tab
    • Add channel button
    • three more filters (for now)

Client procurement process

  • Biz process: depends on timing & budget
  • Implementation
    • a few weeks, already very impressive
  • What can be slow: Data availability
    • departments refuse to or are slow to give you data
      Summary
  • From biz process to actual deployment, it can be months.

Demo

  • Who to demo
    • Telus architect, can give early feedback
    • TD Architect and one other, they need fully working system
  • Demo highlights
    • front end
    • back end, able to run MQL, how to import data
    • Internationalization / No Chinese -> Not finished yet
  • Geoffrey okay to get early feedbacks

Day 30: Separate out the bootstrap vue table column picker

  • Tricky to write tests for it as it needs static: true to assert the rendering

Day 32: Listen to indie hackers again: Be truly helpful to earn money

What I learned

  • Think of what users need
  • Write it down in Github is truly helpful to learning. I don't actually learn until I write it down in my own words
  • When you go deep into something, all the articles are shallow, and I'm lonely in that land.

Being indie hackers, you have the time to learn the skills, and then be truly helpful for a while. You have to make profits years later.

How do you reconcile the work I'm doing as a contractor?

  • Corporate-y work, realizing my value, but the thing I built doesn't belong to me.
  • I don't own the time, but someone else is paying for it.
    • If I don't work, I don't get paid => financial freedom
    • I don't get to control the product. Someone else has to tell me what to do. => development freedom

I already have a cushy job. I'm not motivated to work that hard. What to do?

Day 33: 听完有质感的播客:蒋方舟的描述

30岁以后,这个叫天赋的账号,已经处于破产的状态,我们不能从里面取钱了。需要其他的燃料去支撑自己走下去。

自己写的东西,有时候不会自己去看

有机会就及时记录下来,将来的自己不会再有机会去经历当时的心境和社会状态了。

感觉:听播客是要一种温度

  • 温度,促使自己更好,减少孤独感
  • 知识,得到更多知识,更 resourceful

Day 34: Come back to argos: Deploy to wecise platform

  • How to set the asset path for VueJS?

    • update vue.config.js and add public path setting
  • Integrate early, and make all the lines of code count

    • done. URL is:
http://18.188.85.82:8080/fs/app/argos/index.html?type=open&issys=true

I can only do preview, and it renders the API successfully.

Wire up wecise events API

  • test out the API / MQL, and see what we can get back
    • No data, so he didn't import the data there
  • how to call APIs from front end? scripts?

Let's get back to UI work

  • Change side panel to be tabs (update apps to services)
  • Make search to be tags
  • Update right click context menu to be closer to the mockup
  • Get Performance chart to work with node filter
  • Get Performance chart to work with time filter
  • Decide on views UI (tabs? or something else)
  • when node filter has no results, the table still shows results
  • Preserve the column width settings in views

Yurui's work

  • Test his branch

Day 35: Finished the 1st day morning habit: 30min hack, rain or shine

  • implemented form tags

Day 36: Day counter should be based on the day I worked on it, not calendar days

  • Performance tab: Run into problems of rendering performance charts
    • No data shown on the chart

Two remaining problems

  • Think about how to resolve the data structure in vuex
  • Global search updates perf chart and table, but node filter doesn't update performance chart
  • When there's no result in the node filter, events table still have data

Finding coding bootcamp for Lilian

Udemy

Ruby on rails

Fundamentals

Python

Day 37: Towards morning hacking habit

Two remaining problems

  • Think about how to resolve the data structure in vuex
  • Global search updates perf chart and table, but node filter doesn't update performance chart
  • When there's no result in the node filter, events table still have data

Performance tab Not sure:

  • drill down/up filter: 5min, hourly, weekly (?)
  • local timeline
  • standardize data
  • how to do baseline
  • Performance tab: stats analysis reports (?)

Day 38: Foam: VSCode + Markdown = Second brain

https://www.geoffreylitt.com/2020/07/19/tools-over-apps-for-personal-notetaking.html

Questions

Things missing:

Foam vs. Github

Good:

  • Local available
  • All markdown based, so it is easily editable by whatever editor. So it is like an org mode

Bad:

  • Not able to upload images Github

And then someone mentions Graph database!

Business opportunity: Publish Foam pages as a service!

  • A way to make money!