Join the conversation

Sign in to join this conversation, and others like it, in the communities you care about.

ZEIT

Our mission is to make cloud computing as easy and accessible as mobile computing. You can find our Next.js community here.

ZEIT / Now

Node.js App with mlab MongoDB

Node.js App with mlab MongoDB

ZEIT / Now · January 11, 2019 at 10:54am (Edited 1 week ago)
I have two questions.
1 ) EDIT: from my understanding are lamdas like microservices but each lamda is stateless and handles one request, right?
https://zeit.co/docs/v2/deployments/concepts/lambdas
So I will need to invoke my MongoDB every time, which how I understood so far can be speed up with this script:
https://mongoosejs.com/docs/lambda.html
See EDITED code-

2) I get a connection error from my DB:
01/11 11:30 AM (10m) DB URL mongodb://<myUsername>:<secretPW>@ds253324.mlab.com:<port>/<db-name> 01/11 11:30 AM (10m) REPORT RequestId: f748ee0f-158b-11e9-81ba-21455d5a886b Duration: 2.58 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 77 MB 01/11 11:35 AM (5m) REPORT RequestId: 9f3583d5-158c-11e9-a02a-bff823a71d3e Duration: 13.15 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 77 MB 01/11 11:35 AM (5m) { MongoNetworkError: failed to connect to server [ds253324.mlab.com:<port>] on first connect [MongoNetworkError: connection 0 to ds253324.mlab.com:<port> timed out] at Pool.<anonymous> (/var/task/user/index.js:39919:11) at emitOne (events.js:116:13) at Pool.emit (events.js:211:7) at Connection.<anonymous> (/var/task/user/index.js:100473:12) at Object.onceWrapper (events.js:317:30) at emitTwo (events.js:126:13) at Connection.emit (events.js:214:7) at Socket.<anonymous> (/var/task/user/index.js:60001:10) at Object.onceWrapper (events.js:313:30) at emitNone (events.js:106:13) at Socket.emit (events.js:208:7) at Socket._onTimeout (net.js:420:8) at ontimeout (timers.js:482:11) at tryOnTimeout (timers.js:317:5) at Timer.listOnTimeout (timers.js:277:5) From previous event: at Promise.longStackTracesCaptureStackTrace [as _captureStackTrace] (/var/task/user/index.js:91793:19) at Promise._resolveFromExecutor (/var/task/user/index.js:90398:10) at new Promise (/var/task/user/index.js:89997:10) at NativeConnection.module.exports.Connection.openUri (/var/task/user/index.js:71395:19) at Mongoose.module.exports.Mongoose.connect (/var/task/user/index.js:95259:15) at setupDB (/var/task/user/index.js:76051:14) at module.exports.async.series (/var/task/user/index.js:76076:17) at /var/task/user/index.js:36038:24 at replenish (/var/task/user/index.js:33169:17) at /var/task/user/index.js:33174:9 at eachOfLimit (/var/task/user/index.js:33199:24) at /var/task/user/index.js:33204:16 at _parallel (/var/task/user/index.js:36037:5) at Object.series (/var/task/user/index.js:36893:5) at Object.<anonymous> (/var/task/user/index.js:76075:7) at __webpack_require__ (/var/task/user/index.js:21:30) at /var/task/user/index.js:85:18 at Object.<anonymous> (/var/task/user/index.js:88:10) at Module._compile (module.js:652:30) at Object.Module._extensions..js (module.js:663:10) at Module.load (module.js:565:32) at tryModuleLoad (module.js:505:12) at Function.Module._load (module.js:497:3) at Module.require (module.js:596:17) at require (internal/module.js:11:18) at Object.<anonymous> (/var/task/launcher.js:16:28) at Module._compile (module.js:652:30) at Object.Module._extensions..js (module.js:663:10) name: 'MongoNetworkError', errorLabels: [ 'TransientTransactionError' ], [Symbol(mongoErrorContextSymbol)]: {} }


Here is my config:
{ "name": "git start backend", "alias": "web.git-start.com", "builds": [ { "src": "/**/*.js", "use": "@now/node-server" } ], "version": 2, "routes": [ { "src": "/(.*)", "dest": "index.js" } // It the same conn problem without routes or with { "src": "/(.*)", "dest": "/$1" } ], "env": { "NODE_ENV": "production", ... "MongoDBUrl":"@gs_mongo_db_url" } }
My deployment
...sh/_src
shows me all files required by the app in Output.

The console log in my index.js
require('dotenv').config(); const async = require('async'); const debugBoot = require('debug')('boot'); // app dep const express = require('express'); const bodyParser = require('body-parser'); //db dep const Promise = require("bluebird"); const mongoose = require('mongoose'); // db fn const setupDB = (callback) => { debugBoot("Setting up DB"); if(!process.env.MongoDBUrl) return callback("Please set MongoDBUrl"); connectOptions = { useNewUrlParser: true }; Promise.config({ longStackTraces: true, warnings: { wForgottenReturn: false } }); mongoose.Promise = Promise; mongoose.set('error', true); console.log("DB URL",encodeURI(process.env.MongoDBUrl)); mongoose.connect(encodeURI(process.env.MongoDBUrl), connectOptions, (err) => { if (err) return callback(err); return callback(null); }); }; // app fn const setupServer = (callback) => { debugBoot("Setting up Server"); // main app const app = express(); app.set('port', process.env.PORT || 3000); app.use(bodyParser.json()); app.use(bodyParser.urlencoded({extended: true})); require('./routes/routes.js')(app); const server = app.listen(app.get('port'), () => { const port = server.address().port; console.log('Server running at http://localhost:' + port); return callback(); }); }; // =========================================== main =========================================== async.series([ (next) => { setupDB(next); }, (next) => { setupServer(next); } ], (err) => { if(err) { console.log(err); process.exit(1); } });
also shows the correct connection string. I tested the same configuration locally where I can connect to the DB.

EDITED code:
require('dotenv').config(); const async = require('async'); const debugBoot = require('debug')('boot'); // app dep const express = require('express'); const bodyParser = require('body-parser'); //db dep const Promise = require("bluebird"); const mongoose = require('mongoose'); // db fn if(!process.env.MongoDBUrl) { console.log("Please set MongoDBUrl"); process.exit(1); } let conn = null; const mongoDBUri = process.env.MongoDBUrl; console.log("DB URL",encodeURI(mongoDBUri)); Promise.config({ longStackTraces: true, warnings: { wForgottenReturn: false } }); mongoose.Promise = Promise; mongoose.set('error', true); const setupDB = async (context, callback) => { debugBoot("Setting up DB"); //faster db connect, https://mongoosejs.com/docs/lambda.html context.callbackWaitsForEmptyEventLoop = false; if (conn == null) { connectOptions = { bufferCommands: false, // Disable mongoose buffering bufferMaxEntries: 0 // and MongoDB driver buffering }; try { conn = await mongoose.createConnection(mongoDBUri, connectOptions); console.log("Connected to DB"); return callback(null); } catch (err) { console.log("ERR while connecting to DB", err); return callback("Couldn't connect to MongoDB", err); } } }; // app fn const setupServer = (callback) => { debugBoot("Setting up Server"); // main app const app = express(); app.set('port', process.env.PORT || 3000); app.use(bodyParser.json()); app.use(bodyParser.urlencoded({extended: true})); require('./routes/routes.js')(app); const server = app.listen(app.get('port'), () => { const port = server.address().port; console.log('Server running at http://localhost:' + port); return callback(); }); }; // =========================================== main =========================================== exports.handler = function(event, context, callback) { async.series([ (next) => { setupDB(context, next); }, (next) => { setupServer(next); } ], (err) => { if(err) { console.log(err); process.exit(1); } }); };


January 11, 2019 at 2:14pm

January 11, 2019 at 3:23pm

A couple of observations... re: Mongo and specifically mongoose. The example referred to by @paulgdm is fine but I have a few sets of services that use Mongoose and they experienced similar issues to the logs you included. IE: works fine for a while but then exits with an error which doesn't make a whole lot of sense given that mongoose has built-in reconnect/timeout/retry facilities that even when I messed with those parameters seemed to fail to do what they are intended to do, namely reconnect automatically(in the now v2 environment). I have not done a ton of work to figure out why but there are two things to consider: First off is my hacked workaround that has worked consistently = on any mongoose error I do the reconnect myself instead of depending on mongooses built-in facility. The second is taking a look at the Mongoose docs regarding running under the AWS lambda environment where it shows turning all that automatic connection processing off/command buffering etc.. https://mongoosejs.com/docs/lambda.html

  • reply
  • like
//snippet showing what worked for me in forcing Mongoose to reconnect in now v2

//The following DOES NOT WORK in now v2 standalone.
//Every once in a while it will show logs like yours
//Afer that log signature it will NEVER reconnect.

function connect(){
  mongoose.connect(database, { useNewUrlParser: true, connectTimeoutMS: 10000, auto_reconnect: true }).catch(err => console.log(err));
}

//Here is my hack-y work around that DOES WORK.
//The key part is the db.on('disconnected'...
//And db.on('error'...
//IE explicit disconnect then explicit connect.

//The rest is just so I could get a sense of what exatly is happening 

const db = mongoose.connection ;

db.on('connecting', () => {
  console.info('Connecting to MongoDB...');
});

db.on('error', (error) => {
  console.error(`MongoDB connection error: ${error}`);
  mongoose.disconnect();
});

db.on('connected', () => {
  console.info('Connected to MongoDB!');
});

db.once('open', () => {
  console.info('MongoDB connection opened!');
});

db.on('reconnected', () => {
  console.info('MongoDB reconnected!');
});

db.on('disconnected', () => {
  console.error(`MongoDB disconnected! Reconnecting in ${reconnect_timeout / 1000}s...`);
  setTimeout(() => connect(), reconnect_timeout);
});

connect();

module.exports = mongoose;



like-fill
2
  • reply
  • like

One last thing... these servers just happen to use MLAB

  • reply
  • like

January 11, 2019 at 4:44pm

Hi thx for the replies. Actually, as mentioned in the EDIT I looked at the docs at Zeit and mongoose for using the lambda fn. Unfortunately, I'm still stuck on both questions:

1) I connect the db and listen on a port by executing the handler:

exports.handler = function(event, context) {
    async.series([
        (next) => { setupDB(context, next); },
        (next) => { setupServer(next); }
    ], (err) => {
        if(err) {
            console.log(err);
            process.exit(1);
        }
    });

But the handler is never called, just the index.js is executed. I don't really see in the docs: https://zeit.co/docs/v2/deployments/concepts/lambdas

how to access the event and context variable. This variable seems to be required to reuse the connection: https://mongoosejs.com/docs/lambda.html

2)

So at the moment it doesn't fail anymore, but just because it doesn't connect anymore. I think the second issue, with the connection is still persistent. => I will test Roberts's solution when I get 1) running again (with context & event).

Edited
  • reply
  • like

Okay I think you have two issues I was only commenting on the mongoose issue itself which I have experience in the v2 environment with both the native lambda env as well as the zeit/node-server builder which was the database becoming disconnected every once in a while then never reconnecting again unless I explicitly did my own disconnect on error and reconnect (IE the built-in stuff does not seem to work for mongoose). The other issue is I think that the node-server builder is lambda-izing the entire express server which I have found extremely problematic for just about everything I have tried that is non-trivial... I don't have time to look thru all your code today or even a little bit but "event/context" are not something you are being handed by the now v2 environment they are abstractions being created by code inside the lambda one way or another... take a look at node-server builder, it will show you exactly what's going on prior to getting to your index.js. Personally I have decided NOT to use express inside nowV2 via node-server. It's just as easy/more reliable to port all of my routes to native zeit/node builder and chuck out express by using the platform as my router (I actually stopped using express(and things like it) a long time ago and only have legacy stuff in it anyway).

  • reply
  • like

January 11, 2019 at 11:00pm

a) Ok, I can try the reconnect if I understood the rest but it seemed like it didn't even do the first connection properly.

b) My idea was to keep the lambdas small but use them more as Microservices, an own lambda for every route (how its probably correctly done) is for my simple task a bit over engineered imho.

c) If I'm not handed event and context how can I keep the connection between different lamda units, so they don't start cold? From the mongoose blog I understood that this must be set:

context.callbackWaitsForEmptyEventLoop = false;

d) " take a look at node-server builder", you mean the logs in ...now.sh while building the app?

Edited
  • reply
  • like

a) Ok, I can try the reconnect if I understood the rest but it seemed like it didn't even do the first connection properly.

b) My idea was to keep the lambdas small but use them more as Microservices, an own lambda for every route (how its probably correctly done) is for my simple task a bit over engineered imho.

c) If I'm not handed event and context how can I keep the connection between different lamda units, so they don't start cold? From the mongoose blog I understood that this must be set:

context.callbackWaitsForEmptyEventLoop = false;

d) " take a look at node-server builder", you mean the logs in ...now.sh while building the app?

a) from my investigations one of three things happens with mongoose (and I happened to be running with mlab at the time I was looking at failure conditions that are similar to the log signature you provided) 1: for whatever reason that first connection does not work and then NEVER gets attempted again unless you do it explicitly and potentially try again with the delay. 2: After some indeterminate amount of time of it stops working with again it only trying once but will not work without the explicit disconnect (probably after a cold start) 3: there's actually a problem with connectivity in which you will see repeated attempts in the log if you code it to spin with attempts every X number of seconds.

Edited
  • reply
  • like

a) Ok, I can try the reconnect if I understood the rest but it seemed like it didn't even do the first connection properly.

b) My idea was to keep the lambdas small but use them more as Microservices, an own lambda for every route (how its probably correctly done) is for my simple task a bit over engineered imho.

c) If I'm not handed event and context how can I keep the connection between different lamda units, so they don't start cold? From the mongoose blog I understood that this must be set:

context.callbackWaitsForEmptyEventLoop = false;

d) " take a look at node-server builder", you mean the logs in ...now.sh while building the app?

b) not really as you can package up a crap ton of micro services under one deployment so it really doesn't matter too much where the dispatch is happening as long as those services are not merely calling each other (IE the business logic is actually in the front end manipulating many many services on the backend for no good reason... if this is the case some of those "routes" you may want to take a look at and package up as function calls witch will ABSOLUTELY share an invocation and db connection etc etc... as an aside you cannot depend on any given service invocation sharing anything even if in the same deployment in a lambda environment that's actually the whole point. At a small scale with few invocations this potentially can be extremely "wasteful" IE every single service call at low frequency could theoretically be instantiated from scratch DB connection etc on completely different physical hardware in different data centers which is ridiculous on a micro level IE one browser client calling 5 services packaged together potentially firing up a cold start on 5 different machines, 5 different data centers each one being brought down in between. (shouldn't happen but could - that's up to the infrastructure to manage) on a macro scale of who cares and there's already a dozen instances of each package and say 10 services each all completely independent to serve a large work load who cares..., each will be sharing their own connection across various client invocations. So at large scale it' better and far more fine grained and at small scale it "wastes" resource at an amount that's inconsequential. The upside is compared to an entire system image each lambda is light weight enough so that a cold start is extremely fast. (Probably why Zeit trying to run docker images that need lots of cold boots ummm didn't work out and they are going to v2 in the first place)

Edited
  • reply
  • like

a) Ok, I can try the reconnect if I understood the rest but it seemed like it didn't even do the first connection properly.

b) My idea was to keep the lambdas small but use them more as Microservices, an own lambda for every route (how its probably correctly done) is for my simple task a bit over engineered imho.

c) If I'm not handed event and context how can I keep the connection between different lamda units, so they don't start cold? From the mongoose blog I understood that this must be set:

context.callbackWaitsForEmptyEventLoop = false;

d) " take a look at node-server builder", you mean the logs in ...now.sh while building the app?

c - see items A + B = you shouldn't try but realize those are constructed for you in code you are running any way they are not properties of the lambda environment... one way or another those things begin and end in code you control (sort of) inside your instance IE same result as just having a global accessed by various functions within a lambda call...

  • reply
  • like

a) Ok, I can try the reconnect if I understood the rest but it seemed like it didn't even do the first connection properly.

b) My idea was to keep the lambdas small but use them more as Microservices, an own lambda for every route (how its probably correctly done) is for my simple task a bit over engineered imho.

c) If I'm not handed event and context how can I keep the connection between different lamda units, so they don't start cold? From the mongoose blog I understood that this must be set:

context.callbackWaitsForEmptyEventLoop = false;

d) " take a look at node-server builder", you mean the logs in ...now.sh while building the app?

d = I meant look a the @zeit/node-server builder code on git hub to look at what it's doing and why they discourage using it. I personally am not using it and found just taking existing services that were running under express/micro/koa or other dispatcher were easy enough to port to their own lambdas with routes handled by nowV2... lambda environments like AWS lambda+API Gateway, nowv2, stdlib, serverless, etc basically do what express is doing for you anyway... no really great reasons to do that inside the lambda environment. Express doesn't actually do much... in the case of routing nowv2 covers that and more. In the case of things that it doesn't do that express middleware does... well, most of the express "middleware" stuff is super simple wrappers for another node http native package anyway = easy enough for to just call on your own since at the end of the day it's all operating on what the raw zeit/node builder and lambda environment hands you anyway which is... node http req/res objects.

  • reply
  • like

a) from my investigations one of three things happens with mongoose (and I happened to be running with mlab at the time I was looking at failure conditions that are similar to the log signature you provided) 1: for whatever reason that first connection does not work and then NEVER gets attempted again unless you do it explicitly and potentially try again with the delay. 2: After some indeterminate amount of time of it stops working with again it only trying once but will not work without the explicit disconnect (probably after a cold start) 3: there's actually a problem with connectivity in which you will see repeated attempts in the log if you code it to spin with attempts every X number of seconds.

//if you want to reproduce how simple express middleware works you can just do something like this... just write your own wrapper to call whatever modules you want with http req/res which many express middlewares pull that out of ctx and tack on their own crap to the ctx or req or res object etc. Personally I find it easier to see exactly what's going on with explict manipulations anyway rather than really complicated middleware manipulations having had to debug 8,000 express servers ;-)

const middle_crap = require('my-own-wrapper')

module.exports = handle async (req, rep) => { whateveerrr }

const compose = (...fns) => fns.reduce((f, g) => (...args) => f(g(...args)))

module.exports = compose(
    middle_crap,
    etc,
    etc,
    middle_crapn
)(handle)

Edited
  • reply
  • like

January 12, 2019 at 4:19am

Your connection timed out, is this your actual uri?mongodb://<myUsername>:<secretPW>@ds253324.mlab.com:<port>/<db-name>

  • reply
  • like

Your connection timed out, is this your actual uri?mongodb://<myUsername>:<secretPW>@ds253324.mlab.com:<port>/<db-name>

I assume not but if it is that would be a start ;-)

  • reply
  • like

January 12, 2019 at 9:26am

I assume not but if it is that would be a start ;-)

Yes it is (besides the <> fields which I change with the according values) I use the exact same uri to connect from my localhost where it doesn't fail.

Also, if I use a malformed version (locally) e.g. mongodbs:// then I egt an explicit err: MongoParseError: Invalid connection string so I guess that cannot be the problem.

Edited
  • reply
  • like

January 12, 2019 at 11:08am

Also to the context. Here is some more information why they attach the db connection to the context:

"As mentioned in the How It Works page, any Node.js variable that is declared outside the handler method remains initialized across calls, as long as the same container is reused."

https://www.mongodb.com/blog/post/optimizing-aws-lambda-performance-with-mongodb-atlas-and-nodejs

So I'm still wondering how to have those global vars if there is no handler called (just the file). Or is this is an issue of "@now/node-server" and I should instead use "@now/node"? Would then everything outside module.exports be globally available in a hot container?

EDIT: Also I wouldn't be able to use express anymore? Because it would be to much effort to recode a lot of stuff I have and do everything with nodejs's http module:

https://nodejs.org/en/docs/guides/anatomy-of-an-http-transaction/

Edited
  • reply
  • like

January 14, 2019 at 4:39pm

Further thoughts on this from Zeit?

  • reply
  • like

January 14, 2019 at 8:20pm

I've been experiencing the same error. I'm also using mlab with mongoose ande express

  • reply
  • like

I could try @rwboyer solution. The issue is that you will still send 500s errors when the DB is not connected.

  • reply
  • like

I'm curious about the best DB connection + V2 best practices for Mongo and MySQL or Postgres

  • reply
  • like

I'm curious about the best DB connection + V2 best practices for Mongo and MySQL or Postgres

you can catch errors and do/send anything you want.

  • reply
  • like

you can catch errors and do/send anything you want.

True. But I'm seeing this error almost daily, so it seems the connection is very flaky?

  • reply
  • like

if u look carefully the snippetI sent does a disconnect explicitly then reconnects via the on disconnect, this code will not cause a "500" as the error does not bubble up thru to the ultimate node v2 handler. You have to realize that node builder doesn't explicitly send anything only you do but when the equivalent of res.end() is done the lambda is over end of story.

  • reply
  • like

I believe my problem is that I don't fully understand what's happening with the DB connection during the different states of the lambda function.

The connection is stablished when the lambda is turned on, and it should remain open while it's warm.

  • reply
  • like

and when it's cold there should be no connection to mlab. It seems like the error is produced because even though the lambda is warm, it disconnects.

  • reply
  • like

Log in or sign up to chat