How can I retrieve multiple websites asynchronously?

Question

Using the silverlight Windows Phone 8.1 Project

I'm trying to load data from a website. I have to authenticate at that site first however. So I do a post to the Website, using a lightly modified version of the CookieAwareWebClient from here.

class CookieAwareWebClient : WebClient
{
    public CookieContainer Cookies = new CookieContainer();

    protected override WebRequest GetWebRequest(Uri address)
    {
        var request = base.GetWebRequest(address);
        if (request is HttpWebRequest)
            (request as HttpWebRequest).CookieContainer = Cookies;

        return request;
    }
}

Now I make a WebClient send a POST with Username and Password and continue getting my sites async and processing the site data in my DownloadStringCompleted-EventHandler. So far so good. Now I want to expand on that and get multiple websites. I don't have to get them all at once, in fact it would be better to get them one after another.

But I don't know how to go at this.

My code so far:

using System;
using System.Net;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using Windows.Web.Http;
using Microsoft.Phone.Shell;
using StackOverflowApp.Resources;

namespace StackOverflowApp
{
    public partial class MainPage
    {
        private const string URL_DATES = @"/subsite/dates";
        private const string URL_RESULTS = @"/subsite/results";

        private readonly ApplicationBarIconButton btn;

        private int runningOps = 0;
        //Regex's to parse websites
        private readonly Regex regexDates = new Regex(AppResources.regexDates);
        private readonly Regex regexResults = new Regex(AppResources.regexResults);

        private readonly CookieAwareWebClient client = new CookieAwareWebClient();
        private int status;

        // Konstruktor
        public MainPage()
        {
            InitializeComponent();

            btn = ((ApplicationBarIconButton)ApplicationBar.Buttons[0]);

            // = application/x-www-form-urlencoded
            client.Headers[HttpRequestHeader.ContentType] = AppResources.ContentType;
            client.UploadStringCompleted += UploadStringCompleted;
            client.DownloadStringCompleted += DownloadStringCompleted;
        }

        private void GetSite()
        {
            const string POST_STRING = "name={0}&password={1}";

            var settings = new AppSettings();

            if (settings.UsernameSetting.Length < 3 || settings.PasswordSetting.Length < 3)
            {   
                MessageBox.Show(
                    (settings.UsernameSetting.Length < 3 
                        ? "Bitte geben Sie in den Einstellungen einen Benutzernamen ein\r\n" : string.Empty) 
                    +
                    (settings.PasswordSetting.Length < 3
                        ? "Bitte geben Sie in den Einstellungen ein Kennwort ein\r\n" : string.Empty)
                );
                return;
            }


            LoadingBar.IsEnabled = true;     
            LoadingBar.Visibility = Visibility.Visible;
            client.UploadStringAsync(
                new Uri(AppResources.BaseAddress + "subsite/login"),
                "POST",
                string.Format(POST_STRING, settings.UsernameSetting, settings.PasswordSetting));
        }

        private void LoadDates()
        {
            status = 0; //Termine   
            runningOps++;

            client.DownloadStringAsync(new Uri(AppResources.BaseAddress + URL_DATES));
        }

        private void LoadResults()
        {
            status = 1; //Ergebnisse      
            runningOps++;
            client.DownloadStringAsync(new Uri(AppResources.BaseAddress + URL_RESULTS));
        }

        private void DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
        {
            runningOps--;
            if (runningOps == 0)
            {
                //alle Operationen sind fertig 
                LoadingBar.IsEnabled = false;
                LoadingBar.Visibility = Visibility.Collapsed;
                btn.IsEnabled = true;
            }
            if (e.Cancelled || e.Error != null) return;

            //Antwort erhalten
            var source = e.Result.Replace("\r", "").Replace("\n", "");
            switch (status)
            {
                case 0: //Termine geladen

                    foreach (Match match in regexDates.Matches(source))
                    {
                        var tb = new TextBlock();
                        var g = match.Groups;

                        tb.Text = string.Format(
                            "{1} {2} {3}{0}{4} {5}{0}{6}",
                            Environment.NewLine,
                            g[1].Value,
                            g[2].Captures.Count > 0 ? g[2].Value : string.Empty,
                            g[3].Captures.Count > 0 ? "- " + g[3].Value : string.Empty,
                            g[5].Value,
                            g[6].Captures.Count > 0 ? "bei " + g[6].Value : string.Empty,
                            (
                                g[7].Captures.Count > 0
                                    ? g[7].Value 
                                    : string.Empty 
                            )
                                + 
                            (
                                g[8].Captures.Count > 0
                                    ? g[8].Value != g[4].Value
                                        ? g[8].Value + " != " + g[4].Value
                                        : g[8].Value
                                    : g[4].Captures.Count > 0
                                        ? g[4].Value
                                        : string.Empty
                            )
                            );

                        DatesPanel.Children.Add(tb);
                    }

                    break;

                case 1: //Ergebnisse geladen
                    foreach (Match match in regexResults.Matches(source))
                    {
                        var tb = new TextBlock();
                        var g = match.Groups;

                        tb.Text = string.Format(
                            "{1} {2} {3}{0}{4} {5}{0}{6}",
                            Environment.NewLine,
                            g[1].Value,
                            g[2].Captures.Count > 0 ? g[2].Value : string.Empty,
                            g[3].Captures.Count > 0 ? "- " + g[3].Value : string.Empty,
                            g[5].Value,
                            g[6].Captures.Count > 0 ? "bei " + g[6].Value : string.Empty,
                            (
                                g[7].Captures.Count > 0
                                    ? g[7].Value 
                                    : string.Empty 
                            )
                                + 
                            (
                                g[8].Captures.Count > 0
                                    ? g[8].Value != g[4].Value
                                        ? g[8].Value + " != " + g[4].Value
                                        : g[8].Value
                                    : g[4].Captures.Count > 0
                                        ? g[4].Value
                                        : string.Empty
                            )
                            );

                        ResultsPanel.Children.Add(tb);
                    }

                    break;
                default:
                    return;
            }
        }

        void UploadStringCompleted(object sender, UploadStringCompletedEventArgs e)
        {
            //Login completed
            LoadDates();
            //THIS WOULD YIELD AN ERROR FROM THE WEBCLIENT SAYING IT ISNT SUPPORTING MULTIPLE ASYNC ACTIONS
            //LoadResults();
        }

        private async void ClickOnRefresh(object sender, EventArgs e)
        {
            var isUp = await IsUp();
            if (isUp)
                GetSite();
            else                                                         
                MessageBox.Show("Die Seite ist down! :(");

        }

        private void ClickOnSettings(object sender, EventArgs e)
        {
            NavigationService.Navigate(new Uri("/Settings.xaml", UriKind.Relative));
        }

        private async Task<bool> IsUp()
        {
            btn.IsEnabled = false;
            const string ISUPMELINK = "http://www.isup.me/{0}";
            var data = await RequestData(string.Format(ISUPMELINK, AppResources.BaseAddress.Replace("https://", string.Empty)));
            var isUp = !data.Contains("It's not just you!");
            btn.IsEnabled = true;

            return isUp;
        }

        private async void ClickOnTestConnection(object sender, EventArgs e)
        {
            var isUp = await IsUp();
            MessageBox.Show(string.Format("Die Seite ist {0}! :{1}", isUp ? "up" : "down", isUp ? ")" : "("));
        }

        private static async Task<string> RequestData(string url)
        {
            using (var httpClient = new HttpClient())
                return await httpClient.GetStringAsync(new Uri(url));
        }
    }
}

What I tryed / What I expected / What did block that:

• My first thought was to let everything happen async and use await on all the requests. So I did my research and found that WebClient has an new async/await implementation. WebClient.DownloadStringTaskAsync however I can't find this method in my WebClient, so I assume there isn't an impementation for WP8.1 atm.

• 2nd idea was to use the HttpClient.GetStringAsync(URI) methode which I use already and which supports async/await. As I said, I need a Cookie to go with the request, so I did my research and found this. However I can't find a HttpClientHandler and also no HttpClient.CookieContainer or equal attributes.

• I also tried waiting for one site to complete and then going to the next, but tbo I blocked my GUI thread and didn't want to struggle with writing the whole eventhandlers in separate threads and I don't know how to do so efficiently

Answer 1

Found a solution.

I can use the WebClient.DownloadStringTaskAsync, when I import the NUGET PACKAGE Microsoft Async. This post made me think of nuget in the first place.