Tuesday, January 14, 2014

WTFisThisRegister, Beginner's Flask App: Dictionary based on Flaskr (uses sqlite3) and deployed to Heroku

Over the course of a night and morning I built a small flask app (Flask is a micro web framework for Python. Django is similar but more heavy-duty). It is essentially a dictionary. You can search for an entry by its keyword, or view the helptexts for all the entries, or (if you are logged in) add and remove entries. It is a straightforward derivative of excellent the Flaskr tutorial.



For me, I am really proud of this app even though it is dirt simple and not much to look at because this is the first time I really interacted with a database in a programming language.

The intent was to make a database of documentation for the AVR Atmega328P microcontroller. However, since then I have decided to move to using a mediawiki (sample page).

The app may be seen live at salty-retreat-5363.herokuapp.com. The username is "admin" and the password is "default". The database gets wiped at least once every 24 hours.

If you'd like to run the app locally and play around with source code, please see https://github.com/nouyang/WTFisThisRegister for detailed instructions. That page also details how to deploy this app (or similar ones, such as Flaskr) to Heroku. I used http://dillinger.io/ to play around with markdown online and learn it and actually document this project with a proper README.


Things Learned


Here are some things I learned in the course of making this:
  1. Gitignore files need to be created before you run "git add ." -- in general, the line telling it to ignore a file needs to exist before you added it to the version control, git won't automagically remove it for you.
  2. Heroku DOES play with sqlite3 but not happily, for instance the database will be wiped at least once every 24 hours. See https://devcenter.heroku.com/articles/sqlite3.
  3. Virtualenv does NOT like spaces in your directory path! This was tricky to debug as even in -v verbose mode the full path for my virtualenv packages was shortened and so I couldn't even notice that my path had spaces in it.
  4. Databases use cursors.


The Process

Here are some images from the process. Click on them to view them in full resolution.

1. First I started out just using a dictionary (no database involved) and a single hello.py file.

2. I then followed the Flaskr tutorial. After doing so I wanted to modify it so that the text was monospaced and preserved whitespaces, because it doesn't by default.


I used the "pre" tag at first.


but soon switched to using CSS.
This actually caused me a lot of headache later troubleshooting why there was random whitespace in my design, because I didn't restrict where the CSS was applied enough. So much whitespace:


 In the end I had this file: https://github.com/nouyang/WTFisThisRegister/blob/master/static/style.css
.helptext { preserve whitespace with wrapping CSS }
and the corresponding code in the html file was 
 <div class="helptext"%gt;
{{ entry.helptext|safe }}</div>

3. I wanted to be able to delete entries, so I added a delete button.
In show_entries.html oops can't show it because blogger can't escape this HTML properly :/ In the main python file: 
@app.route('/delete', methods='POST'])                                                                                         
def delete_entry():                                                            
    if not session.get('logged_in'):                                           
        abort(401)                                                             
    g.db.execute('delete from entries where keyword = ?', [request.form['entry_to_delete']])
    g.db.commit()                                                              
    flash('Entry was successfully removed')                                    
    return redirect(url_for('show_entries'))
The "?" is a safe why for sqlite3 to accept text (in this case, whatever was entered in the form with the name entry_to_delete).

4. I learned to play around with the database to test my SQL commands in the terminal:
 $ sqlite3 /tmp/flaskr.db
sqlite> select helptext from entries where helptext='PCMSK0'


5. Finally I added a search functionality to the site:


@app.route('/search', methods=['POST'])                                        
def search_entries():                                                          
    keyword = request.form['searchterm']                                       
    cur = g.db.execute('select helptext from entries where keyword = ?',          
                 [keyword])                                                    
    result = [dict(keyword=keyword, helptext=row[0]) for row in cur.fetchall()]
    return render_template('search.html', result=result)
And set the homepage to be the search page: 

@app.route('/')                                                                
def show_search():                                                             
  return render_template('search.html')

6. I learned about initializing databases, namely by dumbly thinking I needed to include the flaskr.db file and no schema.sql file in the repository to allow other people to get it working on their systems. Actually, I was just missing a line, "init_db()", before "app_run()". In the end I am still keeping it without that line so that I have persistence in the database (I can stop the server and restart it and the database entries will still be there).

All in all I learned a lot! Do browse through the source code and play it with yourself if you'd like.



 =========================================

Liveblog Notes (kept here for reference)


http://flask.pocoo.org/docs/quickstart

http://maximebf.com/blog/2012/10/building-websites-in-python-with-flask/
https://github.com/trtg/flask_assets_tutorial

(did not use: http://www.realpython.com/blog/python/python-web-applications-with-flask-part-ii-app-creation/)

http://flask.pocoo.org/docs/tutorial/introduction/ Ah! Here's the hands-on quickstart.


Hmm, well now the newlines show up but long lines aren't wrapped.
To fix:
http://stackoverflow.com/questions/642140/retain-newlines-in-html-but-wrap-text-possible

.page { width: 70%;
min-width:35em;
border-radius:5px;
}

Modifying the database name: How are you supposed to properly do this? I just deleted /tmp/flaskr.db and then ran in the shell:
sqlite3 /tmp/flaskr.db < schema.sql
and things seemed to work again.

Include a delete button in the future:

If you don't want your data wiped clean, DO NOT do 
$ python
>>> from flaskr import init_db
>>> init_db()

Instead, to play with sqlite3 commands,
sqlite3 /tmp/flaskr.db

And then use the same commands as in flask, but with semicolons.
So to lookup something we need the WHERE command:
e.g. for me it is:
select helptext from entries where keyword=''PCMSK0";

Uhm, okay, now how to display it in my flask app?

ahh what are database i am so confused
http://www.zetcode.com/db/sqlitepythontutorial/
http://docs.python.org/2/library/sqlite3.html

Let's install the AVR eclipse plugin
http://avr-eclipse.sourceforge.net/wiki/index.php/The_AVR_GCC_Toolchain#Debian_and_Ubuntu
http://www.instructables.com/id/How-to-get-started-with-Eclipse-and-AVR/step13/Quick-Tour-Of-Cool-Features/ (windows)

sudo apt-get install eclipse (270+ MB! sigh)
sudo apt-get install gcc-avr binutils-avr gdb-avr avr-libc avrdude

http://avr-eclipse.sourceforge.net/wiki/index.php/Plugin_Download

Heroku deploy
http://flask.pocoo.org/docs/quickstart/#quickstart-deployment
https://devcenter.heroku.com/articles/getting-started-with-python

  1. Install heroku toolchain
  2. Login to heroku
  3. Install and activate virtual environment
  4. Install dependencies (why do I need gunicorn? not sure) 
  5. Make a Procfile
  6. Run foreman and check in browser that app is up on localhost
  7. Download the python .gitignore and add the "venv" line to the top
  8. my own step: add to github, git remote add origin https://github.com/nouyang/WTFisThisRegister.git

https://help.github.com/articles/ignoring-files
OH. The .gitignore file does NOT go in the root directory directly. No wonder everything failed and I just had to delete my github repo and also the .git folder in my root directory.

Instead,
For example, you might create the file at ~/.gitignore_global and add some rules to it. To add this file to your cross-repository configuration, run git config --global core.excludesfile ~/.gitignore_global. Local per-repository rules can be added to the .git/info/exclude file in your repository. This method can be used for locally-generated files that you don't expect other users to generate.

Wait, but what if there's one file, e.g. a passwords for the database file, that I don't want to commit? Maybe it's just messed up because I did the git init and then added the gitignore file or something funky, I don't remember. http://stackoverflow.com/questions/1139762/gitignore-file-not-ignoring

Okay, so now I want to pull the config parameters (passwords, admin usernames, secretkeys) out into a separate file that is gitignored.
Done. Created a file called databaseconfig.py, added the line "from databaseconfig import secretkey, username, password" to the main python file.


Oh I should generate a real secret key
http://stackoverflow.com/questions/18247971/print-python-os-urandom-output-on-terminal
python
>>> import os, binascii
>>> binascii.hexlify(os.urandom(24))

Whoa they actually mean create the file ~/.gitignore_global, not create ~/.gitignore_global/.gitignore
Whoops.

http://stackoverflow.com/questions/4824188/git-ignore-vim-temporary-files
Also, vim and gedit create swap files not covered by github's gitignores, so add
*~
and 
*.swp
*.swo

to ~/.gitignore_global

Yes! Succesfully used .gitignore for once.

(venv)nrw@nrw-PC:~/projects/WTFisThisRegister$ git commit -m "init"
[master (root-commit) 35b1400] init
 10 files changed, 228 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 Procfile
 create mode 100644 README.md
 create mode 100644 WTFisThisRegister.py
 create mode 100644 requirements.txt
 create mode 100644 static/style.css
 create mode 100644 templates/layout.html
 create mode 100644 templates/login.html
 create mode 100644 templates/search.html
 create mode 100644 templates/show_entries.html
Now add it to github.
git remote add origin https://github.com/nouyang/WTFisThisRegister.git
git push -u origin master


Okay, now to finish the heroku deploy.
heroku create 
git push heroku master
Permission denied (publickey).
fatal: The remote end hung up unexpectedly

Whoops I need to add this computer to heroku.
https://devcenter.heroku.com/articles/keys
$ ssh-keygen -t rsa
/home/nrw/.ssh/id_rsa already exists.

Whoops, let's not overwrite that, maybe it will break github. 
cat ~/.ssh/id_rsa.pub

and copy that into heroku under Account (https://dashboard.heroku.com/account).

Alright, let's try pushing to heroku again.
-----> Launching... done, v3
       http://salty-retreat-5363.herokuapp.com deployed to Heroku

Yay!!

Oh noes. It doesn't work on heroku... Is that because I gitignored the databaseconfig.py file and so it doesn't exist on heroku?


Guhuhuh I guess I should learn to use github branches.

Okay nevermind, I found a better answer than having two branches to maintain and update if I ever change anything (bletch).

This gives a clear explanation and example: https://devcenter.heroku.com/articles/config-vars

So in the main python file I put
SECRET_KEY = ENV['SECRETKEY']
and then in the terminal I type
heroku config:set SECRETKEY=blahblah.

Then, git add, git commit, and then git push heroku master.

NOPE that didn't fix it ;____; wahh I am sad.

Let's research....

http://stackoverflow.com/questions/18037027/flask-app-failing-on-heroku-but-working-with-foreman
WHOA good point I forgot to turn debug off! Whaaaa. Okay that is terrible of me.
# configuration
DEBUG = False

WAH. still sad.

Let's try to fix foreman by adding the var to .env... Does that work?

.env does not like spaces around the equal sign assignment operator!
S3_KEY=mykey
NOT 
S3_KEY = mykey

Oh! Make sure to add "import os" to the main python file if we are using os.environ. DUH.
Note: Only today I realized that in pasting URLs from the location bar in chromium-browser (ubuntu 12.10) into blogger, the URLs are automatically turned into hyperlinks, but they remain plain old text and I have to manually turn them into hyperlinks when I copy them from firefox, my main browser :( Such a weird specific bug.

Ah? It still doesn't work?? Why? It works locally on foreman, but not remote on heroku.

Okay, let's look at the heroku logs.
https://devcenter.heroku.com/articles/logging
ImportError: No module named databaseconfig

Whoops. So a gitignore issue.

UGH. Everything becomes 10x more complicated when working in a virtual environment. Fine. I'll take the hit and jump full out into non-localhost environment, where I can't just run "python blah.py" I have to run "foreman start" and other issues. Because foreman looks for a ".env" file. But python does not. (http://stackoverflow.com/questions/12335488/cannot-use-environment-variables-for-settings-in-django)

The solution is to use os.environ.get() instead.

Okay, so I'll just write a short if statement checking if os.environ.get() is None. 

SECRET_KEY = os.environ.get('SECRETKEY') #for heroku deploys.                
if SECRET_KEY == None:                                                      
    SECRET_KEY = secretkey    
Great, now my config works in with both "foreman start" and "python WTFisThisRegister.py".

Now to get databaseconfig to show up in heroku. Do I want to?
No. A cleaner solution instead of all this is to use try, importerrors. Since the present of databaseconfig.py can be used to indicate that we are running it locally.


ahhhh now the /entries works locally but there is an internal server error remotely. WHY.
http://stackoverflow.com/questions/8950674/debugging-a-flask-app-running-in-gunicorn
Instead of foreman start, just use foreman run python app.py if you want to debug your application in development.

 Maybe it's an issue with gunicorn or procfile or whatever that stuff is?
 web: gunicorn WTFisThisRegister:app        
http://ryaneshea.com/lightweight-python-apps-with-flask-twitter-bootstrap-and-heroku
changed to
web: python WTFisThisRegister.py

Okay, still works locally but not remotely. Maybe it is a port issue. Sucks that these tiny changes debugging production (heroku environment) take forever to try (30 seconds for each time I git push heroku master)

2014-01-12T05:58:48.956357+00:00 heroku[web.1]: State changed from crashed to starting
Yay! Fixed ports issue.

AGH. Still internal server error. WTF.
http://stackoverflow.com/questions/7653010/import-sqlite3-with-python2-7-on-heroku
So sqlite3 could NOT be used on the previous version of heroku. However, I think it should be able to now?
https://devcenter.heroku.com/articles/dynos#ephemeral-filesystem
http://stackoverflow.com/questions/7784471/using-sqlite3-on-heroku-cedar-stack 
CANNOT. Ugh. So this is all pointless. Time to migrate to postgreSQL or something that supports sqlite3.

Sigh. I guess I'll just submit a documentation issue about this whole fiasco.
https://github.com/mitsuhiko/flask/issues/949

Okay, looking into how other people deployed their flaskr projects.
https://github.com/mjhea0/flaskr-tdd
This person has a LOT of documentation and talks about jquery / AJAX. Todo for the future!

https://github.com/mjhea0/flaskr-tdd/blob/master/app.py
WAIT. WTF. It looks like you can use sqlite3, I'm just dumb and you have to include the databse in your root directly, not off in /tmp/ somewhere.

So, looks like DATABASE = 'flaskr.db'
sqlite3 ./flaskr.db < schema.sql

Note that the way I have it set up -- foreman locally will run with Debug = False, heroku will run with Debug = false, and python locally will run with Debug = True. A rather peculiar state of affairs, but fine for me.


Let's debug the download-from-github install for the Readme.
$ virtualenv --distribute --no-site-packages venv
The --no-site-packages flag is deprecated; it is now the default behavior.
New python executable in env/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...
  Error [Errno 2] No such file or directory while executing command /home/nrw/projects/t...env/bin/easy_install /usr/share/python-vi...p-1.1.debian1.tar.gz
...Installing pip...done.

Installing existing pip-1.1.debian1.tar.gz distribution: /usr/share/python-virtualenv/pip-1.1.debian1.tar.gz
fails. 

Nope, I'm using --distribute and it fails.


 http://stackoverflow.com/questions/10068388/virtualenv-returning-a-no-such-file-or-directory-error
sudo apt-get remove python-virtualenv
sudo pip install virtualenv

bash: /usr/bin/virtualenv: No such file or directory

Sigh.
sudo pip uninstall virtualenv
sudo apt-get install python-virtualenv

NOPE still get the error.
Spaces?

UGH. It's because my path had spaces.

Posted by nrobot at Tuesday, January 14, 2014

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)