Articles from 1 to 5
I want a new architecture for the web. But what web ?
The Traditional Web
A Web Site is a collection of Web Pages. A Web Page is a document, a static document I might add. The document contains text, images, sound and videos to present some information to the public. This is the traditional version of the web that I respect very much.
There is no need to change this as it works very well. The web was designed to allow a collection of documents to be accessed and does the job very well
The one this that is sensible to do however is not to use any scripting language to render the content of the pages. By definitions, these pages are static, using a scripting language like PHP is not only going to be inefficient, but is going to be a security risk.
This is an entire different thing, and the way we do this is completely wrong. We think that a web application is a collection of dynamic web pages. This should be erased from your mind.
A Web Application is an application of its own right.
Most unfortunately, applications are written in HTML which is not suited for this purpose. The widgets are very few and the framework is not suited to create complex User Interfaces.
The very bad thing to do
You should not do this. Unfortunately, this is how it works for most.
The sensible thing to do
Please do this!!!
An application is an application. It's a bundle and can't be separated into pages. It can be separated into sub-sections if you want, but those will necessarily trigger a page refresh.
The application should be the equal of a desktop application using a native toolkit, except that web technologies are used instead. Contacting the server should be limited to what is necessary only (fetching resources, exchange data, ...). In particular, the templates should be on the client side.
On the server side, you'd have just a classic Web API, and static resources. The Web API should be designed with care and security in mind. It should be easily shared with third parties that want to integrate within your application.
It is as simple as that:
- Server = Web API + Resources
- Client = UI (incl templates)
I heard that SproutCore is a solution to work this way, but it was horrible to use it as well.
Perhaps, we need to get away from frameworks, there are many solutions that can integrate well together and that don't need a framework. Read this article for example: Grab UI by the Handlebar
I have a very simple example to show you what I mean, the comment
look at the sources) that will add the comment features. The page
delivered by the server contain no information about the comments,
they miss the comments.
The script does an AJAX query on a third party server (with PHP and a database). The arguments are the URL of the page and the answer is the JSON formatted list of comments for the page. These comments are then presented on the page. (The refresh link does this again)
The script also creates a form to add a comment. When the form is submitted, an AJAX query is made with the URL of the page and the content of the comment as arguments.
This is how all web applications should work.
Going further: static databases on the server
I already wrote an article on this, but I'll summarize the idea here.
With this new idea for web applications, the application logic is moved from the server to the client. Perhaps not all the way, but for simple applications like the comments above, the server API is nothing much than a fancy database API.
Why not model a database on this model? But it already exists: CouchDB. This is a database which API is a web API. I want to take the principles of this database and mix them with the ideas of a static page generator.
The idea is that read only access when the resource URL is known, a static page should correspond to the URL and a classic web server should answer the query.
Only update and search queries would be filtered by the web server to a FastCGI application that is going to update the static files, or look at them to answer the search.
I find it difficult to find advantages of this compared to CouchDB. There is one thing, your data will always be accessible read only. No obscure database format that require a database server that might not work on new hardware, might not be maintained any more...
With this approach, most web applications could be composed of static assets that are accessing a database with a Web API. The static assets and the database could be on the same server or on a different one. No limit is posed here, except the same origin restriction of web browsers.
Fortunately, a new standard (that I'm using for the comments of this website) let you specify if the same origin policy should apply or not: Cross-Origin Resource Sharing
What about Cookies Then?
They should not exist in their current form.
This was a late addition to the HTTP standard, because it was already implemented. It was never really thought of, and is a breach of the original HTTP model.
What are XSS?
Cross-site scripting (XSS) is a type of computer security vulnerability typically found in Web applications, such as web browsers through breaches of browser security, that enables attackers to inject client-side script into Web pages viewed by other users. A cross-site scripting vulnerability may be used by attackers to bypass access controls such as the same origin policy. Cross-site scripting carried out on websites accounted for roughly 84% of all security vulnerabilities documented by Symantec as of 2007. Their effect may range from a petty nuisance to a significant security risk, depending on the sensitivity of the data handled by the vulnerable site and the nature of any security mitigation implemented by the site's owner.
This is a wrong way to look at the problem. Inclusion of other items in web pages dates from the very beginning. It is the foundation of the web. What allows to have rich documents, with links between them. The same origin policy is just one of these walls that I talked about, that assumes that all pages on the current domain can be trusted. Is that true?
The real story is that if the design of the web was correct, the script injected by attackers in web pages would be completely harmless. The same origin policy shouldn't exist and bypassing it should be by no mean be a security risk.
We have come to a situation where WebGL can be a security risk because it can determine the colours of an image included in the web page. Just because an image can, from the very beginning of the web, bypass the same origin policy. Some people say that images should respect the same origin policy, but what they don't imagine is that we could just forget all of this mess.
The Real Problem.
The real problem is that when loading foreign content, though AJAX, the use of images or whatever, a page can load a foreign page where you are logged in using cookies. And access your foreign account. For example your bank account.
The problem is that the page containing the XSS script was granted the access to the cookie protecting your bank account.
Why the hell your bank account is just protected by a cookie than any page can have access to???
Unless you have in your address bar the address of your bank, the cookie protecting your bank account should be locked away. In fact, instead of having everything in a page respect the same origin policy, cookies should be the single thing respecting the same origin policy;
This is just a change in the handling of the cookie by the browser. Not a very big change at that. Only cookies for the domain in the address bar should be exploitable. But it would break a number of assumption:
Third party cookies would no longer exist, this would be a big shock for advertisers and tracking programs. A loss for these companies, but a win for personal privacy.
Allow third party cookies nonetheless
We could nonetheless allow third party cookies, but only in the context of a specific page:
- each tab/window of the browser share the same cookie jar for the requests
- for each tab/window, a separate cookie jar would be assigned that will be emptied when a link is followed to a different origin
- the sub-elements of a page that are not of the same origin of the page itself, should have their cookies in the sub cookie jar only
- if there are sub-sub elements, they should be assigned to a sub-sub cookie jar recursively.
Allow third party cookies to access the main cookie jar on demand
This would nonetheless break a number of things. Third party scripts expect to see the same cookies they set a while ago, even if it was not on the same page.
A solution to this problem would be to add an option in the browser UI to specifically allow a third party script in a page to access the global cookie jar. There would be a button saying for example "Some cookies have been blocked". When clicked it would open a menu with:
- Allow access to ad.doubleclick.com
- Allow access to facebook.com
- Allow access to my-bank.com
A sensible person would not allow a blog with an XSS script access his bank website, but would allow facebook if he wants to use a facebook button.
Browser plugins, anyone ?
I'd love a browser plugin for that !
Unfortunately, on Firefox, creating sub cookie jar would be difficult due to the architecture of it. Webkit is better on that.
Cookies aren't necessarily bad, they should just respect the same origin policy they introduced instead of imposing it to all of the other elements of a page. This can already be done now in web browsers, it just needs a consensus. Or at least, an extension with the right UI to bypass the same origin on demand when the scripts needs to.
I want to create a new static webpage generator. There are tons of them, you'll tell me, why do I want to create a new one?
Static Website Generator
Let's see what we have:
- webby: my first love
- nanoc: the second one I used
- webgen: the one I'm using currently (with tons of local modifications)
- jekyll: powering GitHub pages
- and many more most probably
Most of these systems take the assumption that you are a developer and you are comfortable to write code to have your website running. In all of these systems, I was forced to code some Ruby here and there to get it working how I wanted to.
The case of nanoc3 is very special because the main configuration uses a special Ruby DSL to manage the path locations. If you have non technical users, they won't be willing to write code in this file anyway. And for technical users, it might not be powerful enough.
Jekyll is probably the simplest system and can probably be used by non technical users, but it is far too simple and not powerful enough. That's why I didn't used it.
In the end, I modified Webgen a lot to include the following features:
- ability to write ruby code in haml instead on having to rely on the template engine included in webgen
- special .index nodes that will generate a paginated index of articles found dynamically. The index would contain the last articles in reverse order while the pages will contain each N articles in the natural order. This way, an article that end up on page 3 is always going to be on page 3.
- special .tags nodes that will generate .index pages dynamically to create an index or articles for each tag.
If you look around, there are not many static web page generators that permit that. First, I decided I would maintain my own version of webgen with these modifications, but now, I have the idea that the code base is so horrible that I prefer rewrite the same functions from scratch.
Rewriting From Scratch
As I said, I'm not satisfied with the current status of the webgen code. There is a complex system of cache, and a blackboard object that is used to dispatch method call to the correct objects around the system. The problem is that this extra indirection level makes it difficult to know the code path. It would be useful if the hooks in the blackboard would be highly dynamic, but it's mostly static. it serves no purpose whatsoever.
Moreover, I don't like big programs that do everything. And all of these static website generators have a component that is used to determine which pages are out of date, and only update them. This is traditionally what make(1) should do. And recently, I found that redo does the job very well. So, I want it to be an integral part of my new system.
Recently, I wrote a piece of code: WWWSupport. It's a very simple git repository that is supposed to be included as a submodule of the git repository of a website. it contains a daemontools daemon that receives e-mail from a special mailbox and convert them into blog posts on the fly (that's how I'm currently writing this block post).
I want my WWWGen project to integrate the same way into my website.
WWWGen is the name of my website generator. The architecture is very simple:
srcdirectory containing the pages, in the same format as webgen
nodesdirectory containing the files WWWGen is working on to generate output
- An output directory where the result files in
nodesis copied to, and where some static assets are copied as well (using rsync)
- A redo script
all.dothat contains the configuration and invokes the wwwgen script.
The wwwgen script will create the
nodes directory and the redo scripts
necessary to its working in it. Then, it will do three things:
For each source file, the script will create:
- A .node file that contain a relative path to the source file, and represents it.
- As many .outdep files as the source file will generate output files. The .outdep file is not managed by redo (because redo doesn't support multiple targets yet). It references the .node file using a relative path.
Note that during this process, new sources can be generated to allow to create new nodes. This step will be executed until no new sources are generated.
Once this is done, the main script will look for all of the .outdep files and will build the corresponding .out file. The .out file will contain the final processed result of the page
Copy all .out files in the output directory (after removing the .out extension) and all the static files in the static directory.
Note that step 1 and 2 recursively call redo to generate the .node and .out files. This two step design is necessary to account for multiple pages generated from a single source.
In all my projects, I always want to focus on the usability of what I create. I always think that non programmers should be able to do the same that I did, to a certain limit. For example, my personal e-mail server at home is scripted all the way. Reinstalling it should be a matter of:
- installing a debian system with a basic configuration
- clone my git repositories
- Set some basic configuration (hostname, ...)
I agree that even then, not anybody can install at home a mail server, but with a process that simple, it's possible to create user interfaces for it. So even if it's not there, it's a possibility.
I want the same for WWWGen. It leaves a possibility for creating a user interface. Nothing prevents from creating a web application or even a native application, that will create markdown pages with a WYSIWYG editor (à la WordPress). The page files thus created could be checked out in a git repository and pushed to a server. There, a hook will run WWWGen to update the website with the modifications.
This could be seriously a very good alternative to WordPress, and I'd preefer working with such a system than WordPress.
What already exists
I am not very good at creating desktop applications. So I thought I would reuse the existing ones : my mailer. I's like a Content Management System where everything must be configured by hand, and only articles can be submitted using an e-mail.
This post is being sent by e-mail to my personal web server. Currently,
I'm still using plain text with a markdown syntax, but we could reuse
the HTML markup in a mail. This e-mail is then processed by a special
This line was automatically generated by an install.do script. The push.sh script will read the mail using a ruby script and will create a file in my website git repository with the content of the mail. Then webgen is run and the content of the repository is pushed to origin.
As a consequence, the new article appears on the website. This is a very simple form of user interface, but it permits anybody to write blog posts.
What features I would like to see
- Better parsing of HTML e-mail
- Using wwwgen instead of webgen
- Support for image galleries using git-annex in wwwgen
- Support for taking the attached images in an e-mail to create a gallery on my website?
What features a customer would want
A web application that can :
- modify the website configuration in the
- modify any source files, using a WYSIWYG editor when applicable
- add static assets (possibly using git-annex)
- run the website compilation and preview it
- push to a server to update the production website
This is entirely possible.
I just moved the site to webgen. For the occasion, forked the project on github and improved a lot of things. As webgen doesn't seem to be alive, I think I'll be the one maintaining it in the future.
I took the occasion to move my website over to my own web server, hosted at home. If I ever find it unsuitable, I can very easily host my website elsewhere as it is just a bunch of static pages.
More to come ...
What is this all about: web privacy. We are tracked everywhere, and i'd like to help if possible. So, let's design a web browser that for once respects your privacy.
- Each website has its own cookie jar, its own cache and its own HTML5 local storage
- History related css attributes are disabled
- External plugins are only enabled on demand
- Support for Tor/I2P is enabled by default
- You have complete control over who receives what information
- Let's you control in the settings if you want to allow referrers or not.
- The contextual menu let's you open links anonymously (no referrer, anonymous session)
The browser is bundled with a particular UI that let you control everything during your browsing session. it is non intrusive and makes the best choice by default. I am thinking of a notification bar that shows at the bottom. I noticed that this place is not intrusive when I realized that the search bar in Firefox was most of the time open, even if most of the time I didn't use it.
First, let's define a session:
- A session can be used my more than one domain at the same time.
- A session is associated to a specific cache storage
- A session is associated to a specific HTML5 storage
- A session is associated to a specific cookie jar
- A session acn be closed. When it is closed, session cookies are deleted from the session
- A session can be reopened. All long lasting cookies, cache, HTML5 storage and such is then used again.
- A session can be anonymous. In such a case, the session is deleted completely when it is closed.
- A session is associated to none, one or more domains. These domains are the domains the end user can see in the address bar, not the sub items in the page.
Sessions are like Firefox profiles. If you iopen a new session, it's like you opened a new Firefox profile you just created. Because people will never create a different Firefox profile for each site.
If we want to protect privacy, when a link is opened, a new session should be created each time. To make it usable to browse web sites, it is made possible to share sessions in specific cases. Let's define the cases where it might be intelligent to share a profile:
You click a link or submit a form and expect to still be logged-in in the site you are viewing. You don't care if you follow a link to an external page.
User Interface: If the link matches one of the domains of the session, then keep the session. No UI. If the user wanted a new session, the "Open anonymously" entry in the context menu exists. A button on the toolbar might be available to enter a state where we always want to open links anonymously.
If the link points to another domain, then open the link in a new session unless "Open in the same session" was specified in the context menu. The UI contains:
We Protected your privacy by separating <domain of the new site> from the site you were visiting previously (<domain of the previous site>). Choices: [ (1) Create a new anonymous session | ▼ ] | (2) Continue session from <previous domain> | | (3) Use a previous session for <new domain> | | (4) Use session from bookmark "<name>" |
The first choice is considered the default and the page is loaded with it. If the user chooses a new option, then the page is reloaded.
If the user chooses (2), the page is reloaded with the previous session and the user will be asked if "Do you want <old domain> and <new domain> to have access to the same private information about you?". Answers are Yes, No and Always. If the answer is Always, then in the configuration, the two domains are considered one and the same.
The choice (3) will use the most recent session for the new domain. It might be a session currently in use or a session in the history.
There are as many (4) options as there are bookmarks for the new domain. If different bookmarks share a single session, only one bookmark is shown. This choice will load the session from the bookmark.
If (3) and (4) are the same sessions, and there is only one bookmark (4), then the (4) option is left out.
You use a bookmark and expect to continue the session you had for this bookmark (webmails)
The session is simpely stored in the bookmark. When saving a bookmark, there is an option to store the session with it or not.
[X] Do not save any personal information with this bookmark
You open a new URL and you might want reuse a session that was opened for this URL.
The User Interface allows you to restore the session:
We protected your privacy by not sending any personal information to <domain>. If you want <domain> to receive private information, please select: Choices: [ Do not send private information | ▼ ] | Use a previous session for <domain> | | Use session from bookmark "<name>" |
If you can see other use cases, please comment on that.
From these use cases, I can infer three kind of sessions:
- Live sessions, currently in use
- Saved sessions, associated to a bookmark
- Closed sessions in the past, accessible using history. Collected after a too long time.
Now, how to implement that? I was thinking of QtWebKit as I already worked with Qt and it's easy to work with.
- We have a main widget:
QWebView. We want to change the session when a new page is loaded. So we hook up with the signal
- We prevent history related CSS rules by implementing
QWebHistoryInterface, more specifically, we store the history necessary to implement
QWebHistoryInterfacein the session.
- We change the cache by implementing
QAbstractNetworkCacheand setting it using
- We change the cookie jar by implementing
QNetworkCookieJarand setting it using
- Change the local storage path using a directory dedicated for the session
After all that, we'll have to inspect the resulting browser to determine if there are still areas where we fail at protecting privacy.
First postulate: HTML was designed as a stateless protocol
Context: web sites need to maintain a context (or state) to track the client. This is required by the log-in procedures the various websites have. It is also useful to track the user in a web store, to know which items the user wants to buy. In fact, it is requires almost everywhere.
The first solution to be thrown out for this problem are the cookies. People didn't like cookies but now, everyone accepts them. Nothing works without cookies. Why did people dislike cookies back then? They liked their provacy and cookies makes it possible to track the user. Through advertisement networks, the advertiser known exactly which website the user visited. And it is still the case now. What changed is that the users got tried to fight cookies and have every website break, and they got used to it.
People got used to being tracked just as people are used to be watched by video cameras in the street and people are used to get tracked by the government and big companies and banks.
Cookies are a great way to track prople, all because HTTP didn't include session management. The way Google track you is very simple. Google Analytics puts a cookie on your computer and each time you access the Google Server, they know it's the same person. Google is everywhere:
- Many web sites are using Google APIs, or the jQuery library at Google.
- Many web sites ask Google to track their users to know how many prople visit their page.
- Google makes advertisement.
- Youtube, Blogger, Picasa and others are owned by Google
With this alone, Google is found on almost every page. If you have an account at Google (YouTube, Picase, Gmail, Blogger, Android or other), they can even give a name or an e-mail address to all of these information.
Google motto is Don't be Evil, they are perhaps not evil but can they become evil? Yes.
Whatever, my dream HTTP 2.0 protocol would include of course push support like WebSockets, but more importantly: session management. How should this be done?
HTTP and Session Management
When the server needs a session, it initiates the session by giving a session token to the client. The client needs to protect this token from being stolen and should display that a session is in pogress for this website. It could appear on the URL bar for example. The client could close the session at any moment.
With the token, the server provides its validity scope. Domains, subdomains,
path. Only the resources in the session scope will receive the tocken back. If
http://example.com starts a session at
example.com but have an
<iframe> that includes facebook. Facebook won't receive the session token. If
Facebook wants to start a session (because the user wants to log-in) it will
start a second session.
Session cannot escape the page. If you have two tabs open with facebook in each tab (either full page or embedded), the two facebook instances don't share the same session, unless the user explicitely allowed this. For instance, when Facebook starts a session, the browser could tell the user that Facebook already have an existing session and the user would be free to choose between the new session and the existing one.
How does this solve XSS
XSS is when a website you don't trust access the session of a website you trust, and steal it. At least I think so.
With this kind of session management, the session couldn't possibly be stolen. Suppose that the non-trusted site makes an XmlHttpRequest to gmail.com. If cross-domain wasn't forbidden, any web-site could read your mails.
With the new session management, if the untrusted site makes a request to gmail.com, gmail.com session wouldn't be available and the login page would be returned instead of the list of e-mails. If the non trusted website tries to log-in, you would be prompted to associate the Gmail session with the site you don't trust. If you aren't completely idion, you wouldn't allow the online pharmacy to connect to Gmail.
What is known about you? Let's take an average person that uses her credit card, have and Android phone with Gmail, uses Facebook:
- All your relationships are known by Google (Gmail, Google+) and Facebook
- All your interests are known by Google and Facebook (Ad Sense track which website you visit and Facebook have a huge profile on you)
- All your posessions are known to your bank
- Your photograph is known by Google and Facebook (people probably took a photo of you and placed it on their Android phone synchronized with Google)
- Your location is known (using your Android phone, your credit card, or your RFID card you use for public transportation)
If you ever want to keep private, it is becoming very difficult.