sahilrajput03 / learning-monogo-and-mongoosejs

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Readme

Quicks:

Does validation message plays any role when required: false (tldr: no)

// schema:
  password: {
    type: String,
    required: [false, 'Please Enter Your Password'],
    minLength: [8, 'Password should be greater than 8 characters'],
    select: false,
  },
// password field is only retrieved when explicitly asked for like below
const user = await User.findOne({ email }).select('+password')

image

Validatation using mongoose data

const mongoose = require('mongoose');

const userSchema = new mongoose.Schema({
  username: {
    type: String,
    required: true
  },
  email: {
    type: String,
    required: true,
    unique: true
  }
});

const User = mongoose.model('User', userSchema);

// This is useful for simple validation on APIs
async function validateUser() {
  const newUser = new User({
    username: 'john_doe',
    email: 'john.doe@email.com'
  });

  try {
    await newUser.validate();
    console.log('Data is valid!');
  } catch (error) {
    console.error(error.message);
  }
}

validateUser();

Comparison for custom validation libraries for data on backend:

How can a store documents such that I can fetch them in a particular sequence

Query: For e.g., we have a 10 classes in a school, then each class has students with defined roll numbers to them in ascending order. So, we need to fetch list of students in a particular order only, in other words we need to maintain their order/sequence.

Solution:

  1. Use a Sequence Field: Add a sequence field to your documents that represents the order or sequence of the documents. When you retrieve documents, you can use this field to sort them in the desired sequence. When retrieving documents, you can use .sort({ "sequence": 1 }) to get them in ascending order.
    {
      "_id": ObjectId("documentId"),
      "name": "Document Name",
      "sequence": 1
      // Other document fields
    }
  2. Maintain an External List in your class document for e.g., rollNumbers: [studentId1, studentId2, and so on...] that represents the desired sequence. When you need to retrieve documents in a specific order, you can fetch the list of IDs and then retrieve the documents based on this list. This approach allows you to change the order without modifying the documents themselves. However, it requires additional management of the order list.

maxTimeMS (sems like it doesn't work though)

chatGPT

image

    const doc of moviesModel
      .find({ deleted: MovieDeletionStatus.NotDeleted })
      .cursor({ maxTimeMS: 5000 })
  ) {
    // doc
  }

Using .exists() method to check if one or more documents match with a given filter

Note: Model.exists(filter) returns { _id: id_of_first_matched_document } | null.

async podcastUrlIsAvailable(podcastUrl: string): Promise<boolean> {
  const podcastExists = await this.podcastsModel.exists({ rssUrl: podcastUrl });
  return !podcastExists;
}

How many parallel mongodb insert query can I run in parallel with mongoosejs in nodejs?

image

Actively closing mongodb connection to close NestJS app immediately when we run custom scripts

Source: Click here

Problem: When we run a below script with a simple console.log, the app.close works pretty good but when we have some bulky db operations with db the app.close seems to hang out for some time like 1-2 minutes thus we can fix that issue by actively closing the db connection as shown in below screenshot.

image

Sort in mongodb

Short notes:

  1. Default is ascending order ie., sort: 1
  2. Providing sort: -1 to get in descending order.

image

Amazing mongodb query

Link to saved playground by Author: Click here

Stackoverfow Answer: Click here

How can we print more than 20 items in mongo shell?

Source: Click here

image

What is mongodb cluster?

image

Using $in and $nin operator for a single field at the same time

image

Also, $and is redundant and thus we can remove $and and it would work same again:

image

Latitude vs. longitude

Source: Stack Overflow Question: Click here

The latitude must be a number between -90 and 90 and the longitude between -180 and 180.

Find documents within rectangular region specifed by [topRightCoordinate, bottomLeftCoordinates]

Specifies a rectangle for a geospatial $geoWithin query to return documents that are within the bounds of the rectangle, according to their point-based location data.

$box: Click here

{/* using mongocompass's query, worked awesome! */}
db.events.find(
  {
     location: {
       $geoWithin: { $box: [ [topRightCoordinateLat, topRightCoordinateLng], [bottomLeftLat, bottomLeftLng]] }
     }
  }
)

Find documents within a given radius around a given coordinate

const results = await this.eventModel.aggregate([
  {
    $geoNear: {
      near: {
        type: 'Point',
        coordinates: [lattitude, longitude],
      },
      maxDistance: maxDistanceMetres,
      distanceField: 'distance',
      // get distances of each event in miles
      distanceMultiplier: METRES_TO_MILES_MULTIPLIER,
    },
  },
  // any other optional query (NOTE: `$geoNear` must be the first query in the pipeline array of queries
  { $match: query },
]);

GeoJSON Objects and Geospatial Queries in mongodb

Docs:

MongoDB geospatial queries on GeoJSON objects calculate on a sphere; MongoDB uses the WGS84 reference system for geospatial queries on GeoJSON objects.

Syntax:

<field>: { type: <GeoJSON type> , coordinates: <coordinates> }

// Point
{ type: "Point", coordinates: [ 40, 5 ] }

// LineString
{ type: "LineString", coordinates: [ [ 40, 5 ], [ 41, 6 ] ] }

// Polygon
{
  type: "Polygon",
  coordinates: [ [ [ 0 , 0 ] , [ 3 , 6 ] , [ 6 , 1 ] , [ 0 , 0  ] ] ]
}

// Polygons with Multiple Rings
{
  type : "Polygon",
  coordinates : [
     [ [ 0 , 0 ] , [ 3 , 6 ] , [ 6 , 1 ] , [ 0 , 0 ] ],
     [ [ 2 , 2 ] , [ 3 , 3 ] , [ 4 , 2 ] , [ 2 , 2 ] ]
  ]
}


// MultiPoint
{
  type: "MultiPoint",
  coordinates: [
     [ -73.9580, 40.8003 ],
     [ -73.9498, 40.7968 ],
     [ -73.9737, 40.7648 ],
     [ -73.9814, 40.7681 ]
  ]
}

// MultiLineString
{
  type: "MultiLineString",
  coordinates: [
     [ [ -73.96943, 40.78519 ], [ -73.96082, 40.78095 ] ],
     [ [ -73.96415, 40.79229 ], [ -73.95544, 40.78854 ] ],
     [ [ -73.97162, 40.78205 ], [ -73.96374, 40.77715 ] ],
     [ [ -73.97880, 40.77247 ], [ -73.97036, 40.76811 ] ]
  ]
}


// MultiPolygon
{
  type: "MultiPolygon",
  coordinates: [
     [ [ [ -73.958, 40.8003 ], [ -73.9498, 40.7968 ], [ -73.9737, 40.7648 ], [ -73.9814, 40.7681 ], [ -73.958, 40.8003 ] ] ],
     [ [ [ -73.958, 40.8003 ], [ -73.9498, 40.7968 ], [ -73.9737, 40.7648 ], [ -73.958, 40.8003 ] ] ]
  ]
}

// GeometryCollection
{
  type: "GeometryCollection",
  geometries: [
     {
       type: "MultiPoint",
       coordinates: [
          [ -73.9580, 40.8003 ],
          [ -73.9498, 40.7968 ],
          [ -73.9737, 40.7648 ],
          [ -73.9814, 40.7681 ]
       ]
     },
     {
       type: "MultiLineString",
       coordinates: [
          [ [ -73.96943, 40.78519 ], [ -73.96082, 40.78095 ] ],
          [ [ -73.96415, 40.79229 ], [ -73.95544, 40.78854 ] ],
          [ [ -73.97162, 40.78205 ], [ -73.96374, 40.77715 ] ],
          [ [ -73.97880, 40.77247 ], [ -73.97036, 40.76811 ] ]
       ]
     }
  ]
}

Thats how different indexes look in the mongo compass for a collection

image

Typescript with mongoose

Source (official mongoose docs):

    1. Schemas in TypeScript: Click here
    1. Statics and Methods in TypeScript: Click here (below screenshot)

image

Getting all collecion names

// getting names of all models
const mongooseModelsNames = connection.modelNames();

// getting all models as an array
const mongooseModels = connection.modelNames().map((modelName) => connection.model(modelName));

On demand creating an index for a collection

(await connection.db.collections()).find((collection) => collection.collectionName === 'events').createIndex({ location: '2dsphere' });

Using geospatial schema definition example

const citySchema = new mongoose.Schema({
  name: String,
  location: {
    type: {
      type: String,
      enum: ['Point'],
      required: true
    },
    coordinates: {
      type: [Number],
      required: true
    }
  }
});

// to add data use this format:
// loc: { type: "Point", coordinates: [ longitude, latitude ] },

♥ ❣♥ ❣♥ ❣ findOneAndUpdate is best and how to use from the maintainer itself

// creating new document if already doesn't exist (note: we're passing {} as second coz we just want a new document with required fields (search fields in 1st argument)
await this.feedReplyLikeModel.findOneAndUpdate({ feedReplyId, userId }, {}, { upsert: true, new: true });


const doc = await Contact.findOneAndUpdate(
  { phone: request.phone},
  { status: request.status },
  { upsert: true, new: true }
);


# Comment from here: https://stackoverflow.com/a/7486950/10012446
# I don't think you need the {$set: ... } part here as its automatic form my reading
# ~Sahil: People use $set as well:
const doc = await Contact.findOneAndUpdate(
  { phone: request.phone},
  { $set: { status: request.status } },
  { upsert: true, new: true }
);

LEARN: ♥ ❣♥ ❣♥ Difference between upsert: true and new: true. Source: https://stackoverflow.com/a/44794886

They are completely different flags
- If upsert is true then if the row trying to be updated does not exist then a new row is inserted instead , if false then it does not do anything .
- If new is true then the modified document is returned after the update rather than the original , if false then the original document is returned

Usage of $exists

Without $exists:

# Collection
[{"key": 1,car: 10},{"key": 2}]

# Query
db.collection.aggregate([{$match: {car: {$ne: 20}}}])

# Output:
[
  {"_id": ObjectId("5a934e000102030405000000"),"car": 10,"key": 1},
  {"_id": ObjectId("5a934e000102030405000001"),"key": 2}
]

With $exists:

# Collection
[{"key": 1,car: 10},{"key": 2}]

# Query
db.collection.aggregate([{$match: {car: {$exists: true, $ne: 20}}}])

# Output (notice that field which has no `car` key is not returned):
[
  {"_id": ObjectId("5a934e000102030405000000"),"car": 10,"key": 1},
]

We can use $count to count documents easily in aggregate

with mongoose you can do like: const ratingUsersCount = await this.movieUserStatusModel.count({ movieId, rating: { $exists: true, $ne: 0 } });

Source: Mongoplayground

Using $count in aggregate: Docs

Similarly:

  • db.collection.count(): Docs
  • db.collection.estimatedDocumentCount(): Docs
# Documents
[
  {"_id": 400, movieId: 200},
  {"_id": 500,movieId: 200},
  {"_id": 600,movieId: 300}
]

# Query
db.collection.aggregate([
  {"$match": {movieId: 200}},
  {"$count": "totaItems"}
])

# Output
[{"totaItems": 2}]

we can execute multiple queries with using $facet

Using $facet: Click here

But this is a issue that I faced myself and seems there's no such fix for this werid behaviour: Click here

useful updateOne method

Drawback:

  • We don't get created/updated document, so we need fo to make a findOne query explcitly
  • Do not rely on returned value i.e, value.upsertedId as this is only preset if document is created (not returned when its updated)
// its useful as it creates the record with necessary values if it already doesn't exist
await this.movieUserStatusModel.updateOne(
  { movieId, userId },
  { $set: { rating } },
  { upsert: true },
);

Amazing feature of mongoplayground

  1. You can create a shareable link via that button as well:

image

  1. To show output for individual stages

image

  1. We can use only these methods. You can view the entire docs by clicking on the Docs button on the mongo playground.

image

Using $avg for calculating average in a single collection

image

Source: Mongoplayground

Query code:

# collection documents
[
    {"_id": 400,movieId: 200,rating: 1},
    {"_id": 500,movieId: 200,rating: 6},
    {"_id": 600,movieId: 300,rating: 5}
]

# query
db.collection.aggregate([
  // We pass movie._id explicitly for which we want to compute $avg (otherwise it would be joining/lookup/populate for all movie documents which is too costly.
  {$match: {movieId: 200}},
  {$group: {_id: "movieId",average: {$avg: "$rating"}}}
])

Using $avg for calculating average from documents of a different collection

Source:

image

# Template: Multiple Collections
db={
  "movies": [
    {"_id": 100},
    {"_id": 200},
    {"_id": 300}
  ],
  "movieuserstatus": [
    {"_id": 400,movieId: 200,rating: 1},
    {"_id": 500,movieId: 200,rating: 6},
    {"_id": 600,movieId: 300,rating: 5}
  ]
}

# query
db.movies.aggregate([
  // We pass movie._id explicitly for which we want to compute $avg (otherwise it would be joining/lookup/populate for all movie documents which is too costly.
  {"$match": {_id: 200}},
  {$lookup: {from: "movieuserstatus",localField: "_id",foreignField: "movieId",as: "movieuserstatus"}},
  {"$unwind": "$movieuserstatus"},
  {$group: {_id: "$movieuserstatus.movieId",average: {$avg: "$movieuserstatus.rating"}}}
])

dbHasActiveOperations amazing utility funciton used in previous compnay (ssshrr)

Cool!

Run raw mongodb commands with mongoosejs

Source: Click here

image

Deleting all collections or dropping database at once?

import mongoose from 'mongoose'

// TESTED: 6 Dec, 2023 (16.13.1), mongoose: ^7.6.3, mongodb (docker): mongo:4.0.9
// Drop entire database (which will drop all collections in one operation)
await mongoose.connection.dropDatabase();
// Delete all documents from the database
await Promise.all((await connection.db.collections()).map((collection) => collection.deleteMany({})));

currentOp ?

Docs Mongodb: Click here

image

$or: collection scan vs. index scan

Source: Official Docs Mongodb: Click here

image

you can give a simple queryFiler to the populated field's data as well like that

Source: Click here

image

pagination over an array field of a document

// TODO: Remove this comment: [offset=5, limit=5<numberOfItems>] i.e., {$slice: [5, 5]} will bring array i.e., [5,9]
const [feedPost] = await this.feedPostModel.find(
  { _id: new mongoose.Types.ObjectId(id) },
  { likes: { $slice: [offset ?? 0, offset + limit] } },
);

Pagination on array stored in a document field

Source: Click here, another similar here.

image

image

TODO: URGENT: Performance Best Practices: Indexing

Recommened Read by Eric (especially @ ESR rule).

Click here

update() is deprecates and warnings recommends to use updateOne() and updateMany() now!

Read about it here: https://mongoosejs.com/docs/deprecations.html

Only add to a field of mongodb document if doesn't exist already

Source: Click here (~ Credits: Eric)

image

#not_tested You can Enable indexing for fields in backend server directly as well (otherwise we simply enable in the mongodb server In Compass)

image

Directly searching a document of a given _id can be done like that:

image

Thats how we do transactions with mongoose

Transactions are new in MongoDB 4.0 and Mongoose 5.2.0. Transactions let you execute multiple operations in isolation and potentially undo all the operations if one of them fails. This guide will get you started using transactions with Mongoose.

Transactions in Mongoose DOCS: Click here

image

Order of array while searching matters

That means if you use wrong order then the query will not match the document:

image

You don't nee the $in operator to search inside of an array in mongo and mongoose

image

image

find duplicate#1 items in mongodb (finding unique items)

image

Playground Link: mongoplayground.net

# Documents:
[
  { "first_name": "Sahil" },
  { "first_name": "Mohit" },
  { "first_name": "Mohit" },
  { "first_name": "Mandy" },
  { "first_name": "Mandy" },
  { "first_name": "Mandy" },
]

# Query
db.collection.aggregate([
  { "$group": { "_id": "$first_name", "duplicates": { "$sum": 1 } } },
  { "$match": { "_id": { "$ne": null }, "duplicates": { "$gt": 1 } } },
  { "$sort": { "duplicates": -1 } },
  { "$project": { "first_name": "$_id", "_id": 0, duplicates: 1 } }
])
  
# PRO Tip: You can filter some data using a $match stage in the front of the aggregate query as well, for e.g, `{$match: {car: 30}}`

image

find duplicate#2 items in mongodb (finding unique items)

image

Documents:
[
  // good case: both users are deleted
  { "email": "Sahil", "deleted": true }, { "email": "Sahil", "deleted": true },
  // good case: one user deleted, one active
  { "email": "Mohit", "deleted": true }, { "email": "Mohit", "deleted": false },
  // *bad case*: one deleted user, two active user i.e, deleted=false
  { "email": "Mandy", "deleted": true }, { "email": "Mandy", "deleted": false },  { "email": "Mandy", "deleted": false },
]

# Query:
db.collection.aggregate([
  // filter only deleted users
  { $match: { deleted: false, } },
  // now we group by email
  { "$group": {
      "_id": {
        "email": "$email",
        "deleted": "$deleted"// delete is *optional* here though
        
      },
      "duplicates": { "$sum": 1 }
    }
  },
  { "$match": { "_id": { "$ne": null }, "duplicates": { "$gt": 1 } } },
])

Cursor in mongodb, Iterate over documents individualy using stream

Q. (Please see ChatGPT's awesome explanations below as well) Why cursor and not batches (i.e, limit() way)? Ans. Using the skip feature when iterating through results in batches can get slower over time for large data sets, when skip is given a high numeric value, so streaming with a cursor helps to get around that. ~Credits: Eric

image

From ChatGPT:

image

image

How to use updateMany in mongodb/mongoosejs

image

Import and export single collection in mongo using cmd ?

Solution: Click here

# notifications collection
mongoexport --collection=notifications --db=slasher_test --out=notifications.json  -u=root -p=rootpassword --uri=mongodb://localhost:27017 --authenticationDatabase=admin
# users collection
mongoexport --collection=users --db=slasher_test --out=users-mocked.json  -u=root -p=rootpassword --uri=mongodb://localhost:27017 --authenticationDatabase=admin

Sort exceeded memory limit ?

Solution of this problem: Click here

image

stackoverflow queston: Is there a max limit to the $nin operator in MongoDB?

image

whats is $$ROOT

Stack overflow Anser: Click here

You can create mongodb object ids from plain text as well

image

post hooks with mongoose

Docs: Click here

image

Clearing collections (can be used for better test without sideeffects)

import { Connection } from 'mongoose';

export const dropCollections = async (connection: Connection) => {
  await Promise.all(
    (await connection.db.collections()).map((collection) => collection.deleteMany({})),
  );
};

more

Source: Click here

image

Thats how you can hide some of the fields

image

Executing js files with mongodb

// switch to database
use imdb;

// firstly we have to clean up potential remainders
db.dropDatabase();

// create admin user
db.createUser({
user: 'imdb',
pwd: 'simple',
roles: [{ role: 'dbOwner', db:'imdb'}]
});

$within?

This is for checking a point lies in given shape with given values of its edges and coordinates. (Not at all useful to check if a coordinate exists in 10kms range of another coordinate point on earth).

Sourc: Click here

Also, $geoWithin works in similar way. Source: Click here

To share with Eric

Even with this $near operator mongodb doesn't support calculating distance b/w two coordinates. We would need to use some custom logic or some third party lib to get approx distance.

image

image

image

links for eric:

Finding distance between two gps locations (point to point and not the actual travelling distance by roads)

Source: Click here

Also: In case you wanna calculate distance using Haversine formula: Stackoverflow Question

Docs: $geoNear: version4.0, ver6-latest

Docs: Geospatial Queries

I managed to calculate distance b/w two gps locations and returning the docs in nearest first order with just mongo's geospatial utils. ALSO: I tested with my nearby locations as well, it seemed to work for me.

Find this code in mongo-shell-with-watcher folder, yikes! TESTED with my custom gps locations as well. Yikes!!

image

Amazing and sweet explanation of document linking by mongodb

Click here

Maximum number of records in single document in mongodb?

Source: Click here

Docs: MongoDB Limits and Thresholds

image

About


Languages

Language:JavaScript 99.4%Language:Shell 0.6%