Category Archives: web

Flask It

If you have worked around Python and Web technologies then you have heard of Flask. It is well known and there are a lot of examples for getting started with simple apps.

Simple/single apps are great if you are learning, but quickly bog down if you have even a little bit of complexity.

Here are some key things to know before you get too far into a monolithic application.

Modules

Blueprints is the Flask concept for bringing in sub-things or modules. The official documentation is clear and provides an example for rendering a page, but similar concept applies for APIs if your application has no web content.

Our recommendation is, as soon as you have the simple app understood, break it into modules (in your preferred pattern) before adding any more routes or pages.

Exceptions

If you leave the exception handling up to Flask, especially if you have an API only application, you will end up with the base errors being returned, like Method Not Allowed or Server Error and the calling client will have no idea what the cause may be.

To deal with this, you can use exception handlers. Create a MyApplicationException and when that exception occurs, return a json object with meaningful description which your client can then understand. In this Handling Application Errors example, instead of catching HTTPException, catch MyApplicationException, and adjust so that response = HTTPException().get_response() to setup the base exception to send onwards by returning response, e.code.

from flask import json
from werkzeug.exceptions import HTTPException

@app.errorhandler(MyApplicationException)
def handle_exception(e):
    """Return JSON instead of HTML for our application errors."""
    # start with an empty http exception as that is what we wish to return
    response = HTTPException().get_response()
    # replace the body with JSON
    response.data = json.dumps({
        "code": e.code,
        "description": e.description,
        #any other key values you want the client to have access to
    })
    response.content_type = "application/json"
    return response, e.code

Page by Page Angularjs to Angular 2+ Migration

If you have worked with network capable embedded devices, you will have worked with web interfaces to control them as it is a convenient UI interface.

If you have an interface written in AngularJS, then you may be considering updating to Angular as a natural progression.

There are numerous examples on line of controlled and gradual ways to do this, and they seem very valid, but most of them require a consistent architecture on the AngularJS side.

If your main aim is to brute replace routes/pages one by one. i.e. Create a new Angular page to replace the AngularJS one, then caressing AngularJS into Angular ‘style’ is probably not good value for money.

Instead, here is a summary of some sites found which together will allow you to

  1. upgrade your entire AngularJS site into a parent Angular site.
  2. route/re-direct to the base page of your AngularJS site (and effectively hand over the router)
  3. Install a event dispatcher form AngularJS to Angular for information that the new pages need from the old ones.

The first two steps are detailed very nicely and with good detail in this reference: https://medium.com/lapis/running-angularjs-1-6-in-angular-5-side-by-side-d2ed771e2a8f

Doing that will get you an Angular project with your original AngularJS running entirely inside it.

Step 3 is the ‘tunnel’ between the two, and involves a rudimentary dispatcher from AngularJS to Angular.

Remembering that we are deprecating AngularJS, so something simple means we can get to the end result faster.

On the Angular side, in a component.ts of your choosing add.

import { Component, HostListener } from '@angular/core';
---snip---

@Component({
  selector: 'your-component',
  templateUrl: './your.component.html',
  styleUrls: ['./your.component.css']
})
export class YourComponent {

  constructor( ) { }

  @HostListener('window:message', ['$event']) sncListener(data: any) {
    console.log("Your message has arrived, decode the data and act on it");
    if (data.detail.your_msg_code === 'user_deleted') {
       // etc
    }
  }

---snip---
}

The HostListener reference is https://angular.io/api/core/HostListener

The relevant piece here is the ‘window:message’ where window is a system string, as shown in the reference above, and the message is what the AngularJS dispatcher (see further down) has in its CustomEvent.

On the AngularJS side, you need to dispatch the event, as follows.

angular.module('Main', ['ngTouch'])

        .controller('MainController', ['$window', '$scope', '$q', '$http', '$location', '$interval') {
        
---snip---

        $scope.someFunction = function () {
        
          // data here is some variable/object
          yourDispatcher({'your_msg_code': data});

---snip---
        }

        function yourDispatcher(data) {
          // message and detail are part of the custom event so need to be just that.
          var event = new CustomEvent('message', {
	          'detail' : data
	        });
          window.dispatchEvent(event);
        }

---snip---

        
        }

The use of CustomEvent and dispatchEvent was gleaned from here https://stackoverflow.com/questions/41050573/when-angularjs-controller-meets-postmessage-events

and a reference is https://developer.mozilla.org/en-US/docs/Web/API/EventTarget/dispatchEvent and https://developer.mozilla.org/en-US/docs/Web/API/CustomEvent

Finally now you have the means to create new pages in Angular and replace the old ones one by one.

MySQL Monitoring

MySQL is one of the most popular database engines.

Here are some handy sql statements to use to help with your development.

View Processes Connected

To pin point who is connected, list all processes.

SHOW PROCESSLIST;
#idUserHostdbCommandTimeStateInfo
1114testlocalhostDB_ASleep1546
2125testlocalhostQuery0initshow processlist
3126testlocalhostDB_BSleep16
Example list of connected client processes

To filter on fields use the information_schema.

select * from information_schema.processlist where db = 'DB_A'

Monitor Connections During Test

While testing your client software, monitor the connections and see if you are leaking them.

SHOW GLOBAL STATUS WHERE Variable_name RLIKE '^(.connect.)$'

The following variables will give you a quick overview of what is going on.

ConnectionsThe number of connection attempts (successful or not) to the MySQL server.

Max_used_connections
The maximum number of connections that have been in use simultaneously since the server started.
Threads_connectedThe number of currently open connections.

Connections will keep growing as it is all connection that have been made, and will not decrease.

Threads_connected you need to ensure goes back to a low number like one or two (for the connection you are monitoring on). If this stays high once your application ends, you are not closing connections.

Max_used_connections is interesting to see at any one time how many connections are active. This way you can increase the max_connections variable.

Acknowledgements

The following knowledge and help sites provided much of this content.

https://dev.mysql.com/doc/refman/8.0/en/show-status.html

Serving Flask Apps with NGinx and Uwsgi

Even an embedded system is going to need a web application, be it on the embedded device, or a part of a test system.

We recommend flask applications, and serving them via Nginx and uWSGI.

There are a lot of examples online. The following one outlines most of the possibilities you can choose from: deploy-flask-uwsgi-nginx

We recommend using the apps-available and apps-enabled as the method for starting your uwsgi server flask application. This is less intrusive and requires less custom parts like an additional system specific service.

For additional security you could install your applications somewhere other than /var/www if that is being served as your root for nginx (for example use /srv/uwsgi-apps instead).

To ensure independent application environments, us a python virtual environment for each of your applications. If you do choose to use the system libraries, we touch on some issues you may run into.

For the nginx configuration, the default uwsgi socket location for your application can be used. On Ubuntu at least it has been:

uwsgi_pass unix:/run/uwsgi/app/<your-application>/socket;

Trouble Shooting

To help you trouble shoot during setup, once you have defined <yourapplication>.ini file under /etc/uwsgi/apps-available and symlinked to apps-enable, restart uwsgi and then tail the uwsgi log for your application.

sudo systemctl restart uwsgi
sudo tail -f /var/log/uwsgi/app/<yourapplication>.log

If something goes wrong inside your application on (re)starting uwsgi, that log will show it.

For example

- mapped 654912 bytes (639 KB) for 8 cores
- *** Operational MODE: preforking ***
Traceback (most recent call last):
  File "./wsgi.py", line 1, in <module>
    from app import app as application
  File "./app.py", line 1, in <module>
    from flask import Flask, jsonify, request
ModuleNotFoundError: No module named 'flask'
- unable to load app 0 (mountpoint='') (callable not found or import error)

Module Not Found Error: No module name

In the example above either the system (or your virtual environment) does not have flask installed or you are using the wrong python plugin.

If your python –version is 3.6.x the python plugin in your uwsgi .ini file should be

plugin = python36

When you run uwsgi through its apps-enabled/<yourapp>.ini method, the application will not be run as your user, so even if you have installed flask and other libraries using pip(3), they are attached to your user’s local, rather than the system library.

If you install using pip a library you will see the output.

Requirement already satisfied: flask in /home/user/.local/lib/python3.6/site-packages

Note the /home/user/.local.

If you are not using a virtual environment, you need to ensure libraries are system libraries, either install them using their corresponding distribution package (e.g. in Ubuntu, sudo apt install python3-flask, or using sudo -H pip …)

No Python Application Found

If you see the error

--- no python application found, check your startup logs for errors ---

Then look earlier in the application log (/var/log/uwsgi/app/<yourapplication>.log) to see if your application was started or if their was an exception.

Unable to load app

unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***

To decipher this, look into <yourapplication>.ini file for the module and chdir lines

chdir = /srv/uswgi-apps/yourapplication
module = wsgi:myapp

The wsgi of wsgi:myapp must correspond to the file name wsgi.py in your /srv/uwsgi-apps/yourapplication folder and the myapp must correspond to the import line in your wsgi.py file which imports your flask app as ‘myapp‘. In the example here, the flask app file was called app.py (as many tutorials tend to name their app files).

from app import app as myapp

if __name__ == '__main__':
    myapp.run()

Gracefully Reloading Applications

The uWSGI documentations detail how to manage your applications once deployed.

We recommend using at least the touch reload option and for finer grain control the Master FIFO method.

Touch Reload

The touch reload means that any time you change your ini file, that application will be reloaded.

touch-reload = /etc/uwsgi/apps-available/<yourapplication>.ini

DANGER! If using touch reload, verify your .ini changes on a staging server before deploying to a production server, because your application will be down if there is an error loading it.

Master FIFO

You can have one or more master-fifo lines.

master-fifo = /tmp/<yourapplication>-master-fifo

From the uwsgi docs: Only the uid running the master has write access to the fifo.

This means that most likely you need to echo commands to the master fifo as the www-data user (unless you specify a different user using uid in .ini file).

$ sudo -u www-data sh
[sudo] password for user: 
$ echo r > /tmp/<yourapplication>-master-fifo

HUP System Signal

If you already deployed your application and you are not in a position to add touch reload or master fifo, you can use the -HUP signal to reload your application.

sudo kill -HUP $(cat /var/run/uwsgi/app/<yourapplication>/pid)