Giters

junjizhi / junji-blog

My own blog

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Building data apps with Vue

junjizhi opened this issue · comments

commented

Building Data Analytics App with Vue

Introduction

Recently I've been using Vue to build a data analytics web app. The app fetches data from a private API and render them in different forms, e.g., data tables, charts, graphs, and other visualizations.

This is my first project using Vue. After playing with Vue for a few months, I understand why such a well-structured, light-weight web framework becomes popular.

Web frameworks debate aside, I want to talk about some lessons learned about building data analytics apps.

This post is for you if you are:

  • planning to build a data analytics, and
  • evaluating which web framework to use (React vs. Vue), or have picked Vue, and
  • wanting to know what gotchas are ahead.

Server logs analytics app

Without revealing too much details, let's say I was building a server logs analytics app. A screenshot looks like below:

vue-data-app-screenshots

Users can:

  • See the raw logs
  • Group the logs data
  • Show them in tables
  • Visualize nodes in a network topology tab
  • Run SQL queries (for advanced users)

Think of the app as a simplified version of Datadog.

Comparison: Vuetify vs. Bootstrap-Vue vs. other UI libraries

The project would need to render data tables. I found a good article with the following criteria for good data table libraries:

  • Fixed header
  • Horizontal scroll
  • Resizable columns
  • Row style (stripe)
  • Display density
  • Visual Table Summary
  • Pagination
  • Hover Actions
  • Inline Editing
  • Expandable Rows
  • Quick View
  • Modal
  • Sortable Columns
  • Basic Filtering
  • Filter Columns
  • Searchable Columns
  • Add Columns
  • Customizable Columns

When I first started, choosing a UI library was a no-brainer. It's 2021 and rendering basic UI data analytics components, like tables or charts, should have been solved.

I narrowed the choices to BootstrapVue and Vuetify. These two libs check most items in the list above.

In the end I selected BootstrapVue because I worked with Bootstrap before and felt comfortable with the CSS styling naming. Also Bootstrap provides more freedom and space to tinker than Materials design.

Another consideration is, using Vuetify makes the app's look-and-feel too Google-like. I didn't want to get locked-in early on.

Using Bootstrap-Vue

Using a UI library like BootstrapVue sped things up a lot. I could quickly build a basic UI skeleton with stub data and asked for user feedback.

Especially, using b-table was convenient. Most of the controls are supported. It is matter of finding the right props / variants / events for customization or implementing behaviours.

And its documentation is well-organized and useful.

The difficult parts

I ran into some difficulties not long after: It requires work to integrate b-table with Vuex and the backend API.

Let me expand on this.

Most examples given in the b-table docs are using a data() which is assumed available in the component. To fetch data dynamically from the API, we need to use itemsProvider callbacks. I blogged about the trickiness to get itemsProvider worked well with Vuex.

I spent a lot of time to on planning, implementing, and refactoring the code for the following features:

  • searching
  • filtering
  • sorting

For example, when implementing the global search bar, I assume the search results would apply globally. In other words, global search affects all tabs.

To translate into the Vue terms, the table component needs to watch the search input.

As mentioned, b-table manages data by itself, so to make it play nicely with global search, I had to force refresh the table while taking extra care to preserve all current table configurations.

Data live in the front or back end?

During the implementation, this question came up a lot.

In particular, since I used Vuex to share data among components, it is easy to treat Vuex store as a cache.

But with the Vue reactivity design, the UI soon became laggy even though the data array is not so large!

A solution to this problem is make the large data array (which comes from the backend) immutable. Namely, I used Object.freeze(...) before passing the long array to Vuex store.

There are also other best practices to organize the Vuex store logic:

  • Skip get & commit mechanism;
  • Use actions to centralize CRUD
  • (Advanced) Use indexing in Vuex store to improve perf

Overall, it needs some care and planning to get reasonable performance when rendering the data.

Rendering data in charts and other visualization

I picked vue-echarts for rendering the time series data in line or bar charts. These two turned out to be simple and asked most often.

For topology graphs, I used vue-d3-network which is a thin wrapper around d3 in Vue. I picked d3 because it seemed the most mature and powerful graph visualization chart I found. It opens lots of customization opportunities and potentially can be applied to other projects.

Short notes about the back end API

As I mentioned above, I was using a private backend API, which takes care of all data ingestion, modelling, indexing and supporting querying data via API.

These are hard data engineering problems, especially when dealing with large volume of real-time graph data, together with the requirements to support historical data. There's no one size for all solution.

The complexity of dealing with all these problem is out of scope.

Summary

This post talks about my experience of implementing the front end for a data analytics application using Vue.

I used Bootstrap-Vue, Vuex, together with a private API.

During the implementation, I ran into tricky problems for data wrangling tasks like searching / filtering / sorting.

For data visualization, I used vue-echarts and vue-d3-network

Overall, Vue helped me spin up a quick front end scaffold and lays the foundation for future customization.

commented

What's it's good for?

  • Building a front end only app, and talk to backend with API
    • How would it be different from React?
      • Vue is of lighter weight?
      • The state management workflow is simpler
      • Good structuring of data fields, data, computed, watch, method
commented

Using Vue to build a good data table

  • Fast bootstrapping
  • Well structured / classified data, computed, watch, methods fields
  • Bundle and scope html and JS code together
  • Good documentation

Regarding dashboards:

For the end user, the essence of a dashboard is essentially this: a collection of the filtered and carefully curated information on a particular topic, conveyed through visual/graphic instruments for quick ingestion. They don’t care about the subtleties of your data pipeline engineering, or how aesthetic your code is — all they want is a high-level view in 3 seconds. Therefore, our crude application displaying text data means nothing to them, and it’s high time we implement mechanisms to wrap the data with charts.

But Argos is not a dashboard. Below is from this site:

Adopting a detective mindset to study a users’ workflow and find unique opportunities to help them make data-informed decision.

When thinking about the product design, ask what their workflow is.

  • When you develop this chart / dashboard, what would you do with it?

About the data table, how would they use it?

Good Data table must-have

  • Fixed header
  • Horizontal scroll
  • Resizable columns
  • Row style (stripe)
  • Display density
  • Visual Table Summary(?)
  • Pagination
  • Hover Actions (?)
  • Inline Editing
  • Expandable Rows
  • Quick View
  • Modal
  • Sortable Columns
  • Basic Filtering
  • Filter Columns
  • Searchable Columns
  • Add Columns
  • Customizable Columns

Bootstrap vue tables evaluation

  • Fixed header
    • sticky header
  • Horizontal scroll
    • sticky column
  • Resizable columns
  • Row style (stripe)
    • native support
  • Display density
  • Visual Table Summary(?)
  • Pagination
    • native support
  • Hover Actions
    • Need to code by hand
  • Inline Editing
  • Expandable Rows
    • row-details
  • Quick View
  • Modal
  • Sortable Columns
  • Basic Filtering
  • Filter Columns
  • Searchable Columns
  • Add Columns
  • Customizable Columns

Idea: Build a open source data table with BootstrapVue

commented

React vs. Vue

Reference: https://vuejs.org/v2/guide/comparison.html

React wins

  • React has richer ecosystem

Vue wins

  • Vue comes with official packages for state management, routing, or SSR
  • Progressive: Easy to migrate existing apps
  • Higher runtime perf with less tuning effort
  • Scoped CSS, easier to work with
  • Vue CLI: easier to generate skeleton of an app
  • html more familiar to most devs and designers
  • Helpful documentation

    • My own experience

    • Quick to prototype MVP.
    • Overkill to use React
    • **Vue docs are helpful and to the point. ** How did the author know I'm running into the exact problem?
commented

Research on Vue + data analytics

Tips for Data scientists

https://medium.com/@matkowaleczko/web-frameworks-in-analytics-and-data-science-a8b4d9bd7083

  • DS don't know front end
  • frameworks are mostly python
  • Use python + jupyter frameworks can build interactive UI

You may not need fancy visualization, just basic charting is enough

Philosophy question:

  • how much is custom?

Vuetify vs. Bootstrap-Vue vs. other UI libraries

Vuetify is amazing and offers more components out of box than bootstrap-vue

https://medium.com/dailyjs/data-visualization-libraries-for-vue-js-in-2020-c74fb83c5778

  • Good article about classifying apps into 3 types of needs, charts, data table, pivot tables.
  • charting library also has support for calendar

The points to write about:

  • UI framework comes with data table, e.g., Vuetify, Bootstrap-vue, but they are basic features
  • charting lib
    • Zingchart

The difficult parts

The difficult / time-consuming part

  • integrating the controls with different datagrid / charting lib
    • search
    • filter
    • sorting
  • data lives in the front end or back end

Dealing with large arrays in Vuex store

More research

https://forum.vuejs.org/t/vuex-how-to-deal-with-large-store/1755/2
dealing with large data

  • Rely on server side paging
  • Aggressively reset data from front end store

https://medium.com/locale-ai/architecting-vuex-store-for-large-scale-vue-js-applications-24c36137e251
Verbiage: Architecting Vuex store

  • learn new thing: Getters
  • Vuex already supports breaking the store into multiple modules: https://vuex.vuejs.org/guide/modules.html
  • also mentioned resetting vuex store, but it is used when users logging out

Another article about architecting Vuex store:
https://markus.oberlehner.net/blog/should-i-store-this-data-in-vuex/#decision-flow-chart

  • Start with store everything in vuex phase
  • Alternatives: portal-vue; provide/inject (for vue3)

https://reside-ic.github.io/blog/handling-long-arrays-performantly-in-vue.js/
performance tuning for long arrays

  • Object.freeze(..) does the trick, which freezes the API response
    • Can we do that?

Vue + D3

https://medium.com/swlh/modular-data-visualizations-with-vue-js-and-d3-87b37392a589

Mainly solved the problems:

  • how to handle data?
    • data is stored as the local state
  • mostly focused on how to render d3
commented

Research meta

Research on Vue + data analytics (keywords)

  • front end for data analytics
  • Vue analytics dashboard
    • usually Vue for admin dashboard
  • Vue dashboard
  • Vue charting
  • Vue big data
  • Vue data app
    • gets me to Vuetify
  • Vue data visualization
  • React dashboard
  • React analytics
  • React chart

Other Vue resources