The AjaxFileUpload control uses an HTTP Handler named AjaxFileUploadHandler.axd
This handler has the type AjaxControlToolkit.AjaxFileUploadHandler.
You must add this handler to your Web.Config file in order for the AjaxFileUpload control to work.
Here's the Web.Config configuration that you must add:
<system.web>
....
<httpHandlers>
<add verb="*" path="AjaxFileUploadHandler.axd"
type="AjaxControlToolkit.AjaxFileUploadHandler,
AjaxControlToolkit"/>
</httpHandlers>
</system.web>
For IIS7:
<system.webServer>
....
<validation validateIntegratedModeConfiguration="false" />
<handlers>
<add name="AjaxFileUploadHandler" verb="*"
path="AjaxFileUploadHandler.axd"
type="AjaxControlToolkit.AjaxFileUploadHandler,
AjaxControlToolkit"/>
</handlers>
</system.webServer>