The Boschmans Account

Using CherryPy for webform authentication

alex — Sun, 07 Mar 2010 21:48:57 +0000

If you are using CherryPy, I can recommend the webform-based authentication that Arnar Birgisson wrote for ease of use and extensability.

After trying out the included authentication models with CherryPy (I’m using 3.1.2, the last stable version at the moment of writing), I was disappointed in the results. Then I stumbled over a recommandation from someone on Nabble, a web-based programmar’s discussion forum, which pointed to the following wiki page on the CherryPy site:

http://tools.cherrypy.org/wiki/AuthenticationAndAccessRestrictions

The complete program code plus examples are on the page and are well explained.

You can have a skeleton login system (using a hardcoded dictionary) up and running in literally half an hour !

Just copy/paste the code on the page and save it as auth.py in your cherrypy script dir.
Add the hardcoded dictionary containing username and passwords to it (or script the db access, see the example included)
Put ‘require()’ everywhere on your cherrypy pages that need to have login protection – additionally, you can also have roles so that only admins can access certain pages.

Early last week I replaced that hardcoded dictionary and built the db lookup query for the login. Once that was working, I added a ‘my profile’ page to the application I’m working on. Then I thought it would be nice for the admin to have a ‘create user’ form in the admin section to add users. Done that as well, using the jquery-ui to create tabs and seperate content in the admin section.

All in all, a nice week of nice work.

I’m starting to think this might make it’s way to my hosting server one of the coming weeks… although I need to do some more work on showing the user only his keywords and not all the keywords, as well as doing something with the keywords to use them better.

Oh and one more thing: this works better under SSL than in the clear http: sky !

Startup ideas I’m having.

alex — Sun, 28 Feb 2010 18:45:09 +0000

The last few months I’ve been working on various projects, but I haven’t been blogging about them, mostly because I usually only post about either something amusing I found on the Interwebs or because whatever programming problems I’ve just solved merit some attention and I couldn’t readily find a solution for.

This blog has grown from being just a personal blog to one where lately quite a lot of the posts concern programming issues, interspersed with only some personal posts.

Currently I’m following quite a few entrepreneurs out there, and some of these blogs by entrepreneurs talk about start-ups and in the next blog post about a technical issue they had to deal with or solve. I quite like that combination. I think it interesting enough that I will start to blog about my quest to launch a start-up, and how I arrived at that idea.

So why I am looking to set up a start-up ?

Currently I’m working full-time as an Online Sales Manager, but more and more the technical side of things is being done by other people in other departments, less and less by me. Being somebody who loves IT and all the things around it, that leaves me with a serious technical itch to scratch. As pressure at work is sometimes heavy, I’ve started programming again during the evenings as a stress-release. I figure I might ‘do’ something with this programming. Also, this answers my need to do ‘useful’ stuff.

In a large organization a lot of what you do are meetings, meetings, meetings to decide what to do. Then you make decisions which are challenged by committees. Finally after days or weeks or months of wrangling a compromise is reached. And then finally you or someone else can ‘do’ the necessary work. It’s not exactly instant coffee, if you know what I mean. A start-up of one means that I can focus on what I want and do something just about immediately (ahem. We’ll see).

So I’m thinking about setting up a start-up where I can scratch that programming itch and (perhaps) make some pocket money doing that.

I’ve done some soul-searching…

The questions I’ve been asking myself these last 6 months have been various :

What do I want to achieve ?
What do I really want to do ?
Just how far do I want to go with this ?
Do I really want or need a startup ?
What sort of startup ?
And what are my business ideas ?
Do I want to sell something ?
What exactly ?
How much time am I willing to spend on this in time and money ?
And so on…

In the end, I’ve made a few decisions and come to a conclusion.

Decision: I want do this as an independent.

I’m going to set myself up as an independent “on the side”. Even if I fail, if I don’t sell anything or things don’t work out, I want to know what it is to be your own boss, even in a limited way. At least I will have tried, and who knows, I might make make something worthwhile out of it, learn from the experience. I hope to set up something small that can grow over the years.

Decision: Work with the technical strengths I have or can readily acquire.

I have to be realistic. I’ve made a few websites over the past few years, for me or for friends of mine, but I’ve realised that I’m not a real websitedesigner and probably never will be one. I’m no big Photoshop designer that can create a beautiful design from scratch and cut up the psd in elements and then code them in html. I love the user experience of a well-designed site, but have no huge creativity when it comes to designing one myself beyond making it functional and error-free.

I’m more interested in the back-end of the web systems, how things work. Sure, it gives me satisfaction to spruce up a website with JavaScript (JQuery-ui rulez because it’s so easy to make a nice site with it), putting up the layout and such, but I get an even bigger kick out of setting up a back-end database and writing the functions that allow you to search it. Making it work, puzzling out the different ways you can access content. Interfacing it with other content.

A combination of HTML, JQuery, Flex and Python, all of which I know to varying degrees allows me to do both the front-end and the back-end of a website in such a way that the result looks reasonably competent.

HTML and JQuery for the front-end
Flex for the display of tables, graphs and complex user interface parts
Python for database connectivity and back-end functions that the front-end parts call.

Web hosting providers that provide Python are currently not that much in evidence, but there are more and more out there. I myself use WebFaction, which I can heartily recommend for their quick answers, nice admin interface and reasonable prices.

That is the base functionality, how to get things done. But that is only a means to an end.

Decision: How much time do I want to invest in it ?

I’m working a full-time job. Do I want to go all-out, quit my job, or do I want to do this part time ? And am I sure that I can control how much time I invest in it ?

After weighing pro’s and contras I’ve decided to do this low-key but long-term. I still want to see my family, not be off to meet some client every evening. I want to build this up over the years.

My main goal, after all, is to scratch my programming itch and make what I create ‘useful’ to other people. So no website design, no consultancy (at least not at first), no excel sheet programming. No individual programming stuff for one particular client (exceptions may happen). I am gonna make things more than one person / company can use. Probably sell them as a service or downloadable program.

And as an entrepreneur I respect told me recently: to keep your costs low, make it yourself. Own it. Don’t sell stuff someone else made if you can create it yourself.

He also told me another golden rule: if you want to do this you full-time you must sell something people want to buy. This seems to be evident, but it’s not. In fact, I’m not sure that I’m going to be doing that. One more reason to start up on the side, testing the waters so to speak.

Decision: What am I going to sell ?

After deciding on limiting my time used for it, I have narrowed the choices down. So I’m selling software or services via a little webshop. What sort of software service ? I’ve decided I’m going to sell tools, tools that I myself think I need. After all, if I can think of uses for those tools, perhaps other can use them as well ? The first one is a social media tool, to keep track of stuff on Twitter. There are already several tools out, but I am not out to corner the market, I out to learn stuff and see if what I make is useful to other people.

Conclusion: I’m doing this. This year.

I’m somebody who needs time to adjust to new stuff. I take my time researching, sometimes too much so. It’s time to start doing.

I’ll update on this in the coming months with how it goes…

Shit you don’t know you don’t know…

alex — Sun, 21 Feb 2010 19:42:59 +0000

Just a pointer to a great blog post about

things you know
things you know you don’t know
things you don’t know you know.

Trust me, go and read the blog post, it’s worth it.

Spoken with Passion

alex — Thu, 11 Feb 2010 20:45:45 +0000

Stephen Fry on (and against) the institution that calls itself the Catholich Church.

The Intelligence² Debate – Stephen Fry (Unedited)

400th blog post !

alex — Sat, 06 Feb 2010 17:25:57 +0000

I just noticed that this is my 400th blog post – I started blogging using Wordpress on Januari 17, 2005 – more than 5 years ago ! Somewhere on the site there’s a link to an earlier blog of mine using Bloxsom, but I quickly switched over to Wordpress.

Wordpress has been – for me – the ideal Content Publishing Management (CMS) system.

400 posts means about 6 blog posts per month on average, something which surprised me – I actually thought it was less.

My first post was about the birthday of Tom, it’s fitting to put a picture of the family as it is now.

A picture of the Boschmans Family end of 2009.

Onwards to another 400 blog posts or another 5 years, whichever comes first !

Simple Python threading.Thread example using Queue

alex — Wed, 03 Feb 2010 22:17:23 +0000

I managed to write a really simple example of using threads in Python that I hope will give more insight on how to adapt my other programming stuff. And re-use this later on, in case I need to revisit this again it would be handy not to scour the internet again to assemble the bits and pieces of threading with Python.

The example below uses 3 threads, and processes 10 pairs of numbers (tuples) that I put in a list.

# Our list of work todo
inputlist_ori = [ (5,5),(10,4),(78,5),(87,2),(65,4),(10,10),(65,2),(88,95),(44,55),(33,3) ]

Those numbers are divided over those 3 threads by the Queue system.

The Queue system itself is limited to 5 slots, although this could easily be changed to more or less. You will notice in the console print that the message “Waiting for threads to finish.” appears after the fifth result, indicating that the queues are being used and the main program has continued on.

After putting everything in the queue system, the program waits for the threads to finish using the .join() function.

All spawned threads keep on being active, running forever, accepting jobs – that is, until the queue is empty, at which point they shut down.

I based most of my simple example on the examples in the Python threading tutorial (.pdf) work of Norman Matloff and Francis Hsu that I referenced before in a previous blog post. However, while their examples undoubtedly do more and are more extensive, they are also more complex. This example is deliberately made as simple as possible so to understand the basic principles of threading and the queue system.

Things I stumbled over:

Duh! You spawn the threads before you fill up the queues with stuff todo…
When printing out things to the console or python shell, things got jumbled because different threads took over from each other – to solve that I used the threading.Lock().acquire() and threading.Lock().release() to make sure that a thread could finish printing. Not sure if I understand completely all the possibilities this offers.
Still a bit stumped on getting more info, name, etc on the thread that is running at the moment – haven’t figured that out yet how to do that.

Feel free to comment and ask questions – if you can improve this program, please let me know !

# threading test
# Alex Boschmans
# www.boschmans.net
# January 2010
 
#
# IMPORT SECTION
#
import threading, Queue
 
#
# Variables setup
#
THREAD_LIMIT = 3                # This is how many threads we want
jobs = Queue.Queue(5)           # This sets up the queue object to use 5 slots
singlelock = threading.Lock()   # This is a lock so threads don’t print trough each other (and other reasons)
 
# Our list of work todo
inputlist_ori = [ (5,5),(10,4),(78,5),(87,2),(65,4),(10,10),(65,2),(88,95),(44,55),(33,3) ]
 
#
# This is called from the main function
# It spawns the threads, fills up the queue with work items that the threads will use
# And then waits for the threads to finish
# This could use some more try:except code…
#
def draadje(inputlist):
print "Inputlist received…"
print inputlist
 
# Spawn the threads
print "Spawning the {0} threads.".format(THREAD_LIMIT)
for x in xrange(THREAD_LIMIT):
print "Thread {0} started.".format(x)
# This is the thread class that we instantiate.
workerbee().start()
 
# Put stuff in queue
print "Putting stuff in queue"
for i in inputlist:
# Block if queue is full, and wait 5 seconds. After 5s raise Queue Full error.
try:
jobs.put(i, block=True, timeout=5)
except:
singlelock.acquire()
print "The queue is full !"
singlelock.release()
 
# Wait for the threads to finish
singlelock.acquire()        # Acquire the lock so we can print
print "Waiting for threads to finish."
singlelock.release()        # Release the lock
jobs.join()                 # This command waits for all threads to finish.
 
#
# Main thread class – based on threading.Thread
# This class is cloned/used as a thread template to spawn those threads.
# The class has a run function that gets a job out of the jobs queue
# And lets the queue object know when it has finished.
#
class workerbee(threading.Thread):
def run(self):
# run forever
while 1:
# Try and get a job out of the queue
try:
job = jobs.get(True,1)
singlelock.acquire()        # Acquire the lock
print "Multiplication of {0} with {1} gives {2}".format(job[0],job[1],(job[0]*job[1]))
singlelock.release()        # Release the lock
# Let the queue know the job is finished.
jobs.task_done()
except:
break           # No more jobs in the queue
 
#
# Executes if the program is started normally, not if imported
#
if __name__ == ‘__main__’:
# Call the mainfunction that sets up threading.
draadje(inputlist_ori)

Sigh. I just finished adding spaces to show where a def ends, and the damn code highlighter removed it again. Grrrrrr. If you want a copy of the code, let me know and I’ll update this post with a zipped copy of it.

Not using regular expressions (re or regex) to find a #hashtag (python).

alex — Wed, 27 Jan 2010 22:01:08 +0000

First, a quick reminder for myself: there’s an extremely good guide to regex on Andrew M. Kuchling’s pages.

Secondly, you don’t really *need* regex to parse for hashtags in a tweet – it’s a bit of overkill. The following code will do as well, and was written in 1 minute after searching 15 minutes in regex how to make certain to include hyphens ( – ) and other non-characters if they are put into the hashtag.

The regular expression that I find works quite well for all hashtags that don’t have a hyphen in it:

>>> hashtag = "This is a #hashtag #test-link #a should#not#work"
>>> x = re.compile(r‘\B#\w+’)
>>> x.findall(hashtag)
[‘#hashtag’, ‘#test’, ‘#a’]

So the above code correctly finds all words beginning with a hashtage, and not the ones that contain a hashtag inside the word. Note that the hyphen and the word after it is not included.

This is the short code I wrote that does all I want:

>>> hashtag = "This is a #hashtag #test-link #a should#not#work"
>>> for word in hashtag.split():
        if word[0] == "#":
                print word              
#hashtag
#test-link
#a

In section 6 of the above-mentioned guide, Andrew states that in some cases string methods (like split) are faster than using regex. For simplicity, I’m going to use the latter code.

Update: Grrr – discovered that the tweets I am processing are in html so have href tags around them – which means ofcourse that there are no blanks for me to split words in. After another unsuccessful session with regex and just to continue I’ve used the BeautifulSoup html parsing library to get around that by stripping out all tags and then splitting the sentence up again. Probably not as efficient as immediately using regex, I’ll have to revisit this in the future.

Using threads in Python

alex — Tue, 26 Jan 2010 22:18:10 +0000

I’ve been trying to setup threading in Python, so that in the back-end of my service system that I’m developing I can query more than one source at the same time. So instead of querying one server and waiting for feedback, I can launch 10 threads and thus query 10 servers and process each server’s feedback via it’s own thread.

So a very vague, generalising definition of a thread is an independent ‘process’ that performs a job that you give it. You can control how many threads that you launch. Each thread is a copy of the original thread that you describe (in essence a python def function that has been wrapped in a thread class).

Right now, my understanding of threads is a bit confused. So far it seems that threading has several different manners of implementing them:

using a number of threads that you launch, use, and forget about them (they go away)
improving on that by putting those threads in a thread pool, and when a thread finishes, re-using it for the next job (so you have 5 threads but 10 jobs to do, those five threads take five jobs, and the first thread that finishes takes on the sixth job, the second thread to finish the seventh job, and so on)
the final step seems to be (I haven’t got that far in my implementation) to set up worker bees that are managed by one thread (a better description is promised, as soon as I have understood it!)

Since I’ve been scouring the net for information over threads, here is a list of resources that discuss, give examples, and explain threads – it’s useful for me to refer to, it might be useful for you as well :

DaveN has an extensive post, with examples, building up gradually. It’s only at the end that you read that the code shown has never been run, which is a bit of a letdown. Still worth a good read though !!
A very thorough 25-page pdf documents that starts from the beginning is available on the site of UC Davis, University of California. It goes into all the nitty gritty details.
An example that uses workers in threads is found on the blog of Danial Taherzadeh.
Another one that discusses using multiple queues chained together can be found on IBM’s developerworks site.
And the blog post from Halotis that started my looking into threads…

Right now I’m using threads in a thread pool, but I’m not doing something right – I noticed that while I have 10 jobs to do, only the first five get done, and the others ‘disappear’.

I guess the only way to get it working is to continue reading the information above until it makes sense. Sometimes I wonder if I’m not slightly masochistic, looking for challenges like that… ai me poor pounding head !

Parrot AR.Drone – instant lust.

alex — Wed, 13 Jan 2010 17:50:04 +0000

Wow. I just saw the advert for a new type of remote controlled helicopter called the Parrot AR.Drone, and I experienced instant want-it syndrome. It’s controlled via your iPhone, it’s not out yet, somewhere in 2010 but from what you see in the demo it is amazing !

It is helicopter on steriods, typically geek stuff, and it looks damn cool.

Excel 2008 for Mac insists on using mm/dd/yyyy instead of dd/mm/yyyy !

alex — Fri, 01 Jan 2010 22:41:57 +0000

I’ve just discovered a really bothersome bug using Excel 2008 for Mac (I’m using the current latest version 12.2.3 of Excel 2008 for Mac). My date columns keep on insisting on using the US format of month-days-year, even when the mac time setting is set to Brussels, Belgium. There does not seem to be a way to fix the setting in Excel, and similar date format problems appear on forums already since Excel 2004.

The workaround I found was to select the date format 01-Mar-2009, in other words show the month as a word instead of as a number. This way I’m sure that the date I’m typing in is correct. But really this is annoying me immensely.

Another workaround (haven’t tried it yet) so that Excel uses the underlying Mac date format is to NOT preselect CELLs and format them as DATES – just type the date in a normal cell using slashes and Excel will then convert the values to the correct international date as set by the mac… I’ll try that next time.

WHY hasn’t this been resolved yet !? What’s so hard about doing this the right way ? I know the answer ofcourse – it’s a big company and everyone there has to jump through 15 hoops simultaneously while hopping on one leg and signing delivery forms with a pen that only works 1 in 3 times