Headers in Puppeteer are not same as in browser

Question

I have this basic post request:

(async () => {

  // Create browser instance, and give it a first tab
    const browser = await puppeteer.launch({headless: false});
    const page = await browser.newPage();

    // Allows you to intercept a request; must appear before
    // your first page.goto()
    await page.setRequestInterception(true);


    // Request intercept handler... will be triggered with 
    // each page.goto() statement
    page.on('request', interceptedRequest => {

        // Here, is where you change the request method and 
        // add your post data
        var data = {
            'method': 'POST',
            'postData': 'HasSell=true&ModelYears=2005&MaxModelYearHidden=2020&Brands=BMW&ModelNames=SERIES+3&TransmissionTypes=Otomatik&FuelTypes=Dizel&CarCases=Sedan&Versions=320D+AUTO&HorsePowers=150&MaxHorsePowerHidden=163&IsDontKnowHorsePowerHidden=0&Km=5.000&IsDontKnowKmHidden=0&OuterDemage-1=1-1&OuterDemage-2=2-1&OuterDemage-3=3-1&OuterDemage-4=4-1&OuterDemage-5=5-1&OuterDemage-6=6-1&OuterDemage-7=7-1&OuterDemage-8=8-1&OuterDemage-9=9-1&OuterDemage-10=10-1&OuterDemage-11=11-1&EquipmentCheckBoxSIS+FARI=SIS+FARI&EquipmentCheckBoxSUNROOF=SUNROOF&EquipmentCheckBoxYOL+BILGISAYARI=YOL+BILGISAYARI&progressValue=10',
            headers: {

            "Content-Type": "application/x-www-form-urlencoded"
          }
       };

        // Request modified... finish sending! 
        interceptedRequest.continue(data);

    });

    // Navigate, trigger the intercept, and resolve the response
    const response = await page.goto('https://www.ikinciyeni.com/fiyatlandirici'); 

    if ((await page.$('#EmailRetail')) !== null) {
    // select and type the email
    await page.click('#EmailRetail')
    await page.keyboard.type('[email protected]')
    // select and type the password
    await page.click('#Password')
    await page.keyboard.type('pass')
    // submit the form
    await page.click('#LoginSubmitBtn')
  }

It goes to the url given and the request is redirected until it lands on login page. It then enters credentials and goes to this url : https://www.ikinciyeni.com/giris

The problem is that it the request made to this page is like this:

POST https://www.ikinciyeni.com/giris HTTP/1.1
Host: www.ikinciyeni.com
Connection: keep-alive
Content-Length: 595
Pragma: no-cache
Cache-Control: no-cache
Content-Type: application/x-www-form-urlencoded
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Referer: https://www.ikinciyeni.com/giris?ReturnUrl=/konsinye-hesap-sonuc&tempId=6c8f8813-fcf7-457d-98ec-287dee173114
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.0 Safari/537.36
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: ASP.NET_SessionId=pggzwsmbsc44c1r3dvilzwtf; Cookie_DimensionId=566878; NSC_xxx.jljodjzfoj.dpn_iuuqt=ffffffffaf1fc83345525d5f4f58455e445a4a423660; __RequestVerificationToken=IWsg8lbQe_Qhm7LFlvOtgaFARnHTCkyo6zc2ZaPOgEQMscTCNVLMjuh3A2y_mSmsGb4yW25qOjbVcucn5LWymmktzdM1

HasSell=true&ModelYears=2005&MaxModelYearHidden=2020&Brands=BMW&ModelNames=SERIES+3&TransmissionTypes=Otomatik&FuelTypes=Dizel&CarCases=Sedan&Versions=320D+AUTO&HorsePowers=150&MaxHorsePowerHidden=163&IsDontKnowHorsePowerHidden=0&Km=5.000&IsDontKnowKmHidden=0&OuterDemage-1=1-1&OuterDemage-2=2-1&OuterDemage-3=3-1&OuterDemage-4=4-1&OuterDemage-5=5-1&OuterDemage-6=6-1&OuterDemage-7=7-1&OuterDemage-8=8-1&OuterDemage-9=9-1&OuterDemage-10=10-1&OuterDemage-11=11-1&EquipmentCheckBoxSIS+FARI=SIS+FARI&EquipmentCheckBoxSUNROOF=SUNROOF&EquipmentCheckBoxYOL+BILGISAYARI=YOL+BILGISAYARI&progressValue=10

whereas when Im making a request using Browser on website itself I get following request

POST https://www.ikinciyeni.com/giris HTTP/1.1
Host: www.ikinciyeni.com
Connection: keep-alive
Content-Length: 296
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
Origin: https://www.ikinciyeni.com
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Referer: https://www.ikinciyeni.com/giris?ReturnUrl=/konsinye-hesap-sonuc&tempId=b58077ab-e293-499c-b232-61d55e242824
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: __gads=ID=b7b47e7d0a959312:T=1588862975:S=ALNI_MYvqBiWkaKMhf4VItO9u99dG6E3Ow; _ga=GA1.2.384301327.1588862976; _hjid=d2ef3b50-8050-4589-8003-7833e3691180; _gid=GA1.2.460781385.1591720353; _fbp=fb.1.1591720354344.887140335; _hjDonePolls=454523; ASP.NET_SessionId=e4npmohe33aj2o2hx444nzj0; __RequestVerificationToken=D9XTJgR45PZvXMdrRB7QgrDPsGK60ghEJqvDeadnr46EXphS0SiP6-knUTOJbFhdI1NYaw5SX-ZB3ZQh8xYXJSfN12U1; Cookie_DimensionId=243006; Cookie_UserType=birey; Cookie_UserRegistration=0; Cookie_ClientType=Birey satici; Cookie_LastClientAction=satis; .WebAuth=B29C833AD3B18C6A7F37E79F444C1089DEEC38F73CED959ABF577D874D38B7A69D63D56FCC04682D72D39A115C991EC2EBB5E6C4401C740110F234EF71D622EABD77A2D8491DEAC4794CB52D3A77B43594A736611DEC6E9C9B06F5E53C9FDFCB26FB3A64; .WebAuth=180228A130674EC2CF4008C0353849E5E069E35C37984336477F7993E1406079028ECDCE9612E109A09192C4992AD15CDA5539F02942C0D9BA8482A7C65025333A1373BDC9CA7711C7832AF4ED204B8BBF558B00; Cookie_UserLastLoginDate=12.6.2020 02:49:08; ins-storage-version=96; NSC_xxx.jljodjzfoj.dpn_iuuqt=ffffffffaf1fc83345525d5f4f58455e445a4a423660; _dc_gtm_UA-41063025-1=1; _dc_gtm_UA-41063025-15=1; _hjAbsoluteSessionInProgress=1

__RequestVerificationToken=h0UJR6bxm6bHC5osMWhTBehedUOMQ47C2MHoHJi6RcHDPCUe-xTm0v5ulEFbo-gUXTxQW9Kphd-HEn_iMQG1yuVaVUI1&Email=emailIentered%40gmail.com&Password=password&lastContractId=0&returnUrl=%2Fkonsinye-hesap-sonuc&hiddenPassword=password&tempId=b58077ab-e293-499c-b232-61d55e242824&webUserId=0

How do I make such a request? How do I append these headers? But these headers are dynamically generated and are unique for each request.

Answer 1

interceptedRequest.headers() is missing some headers. To override or add new headers without missing any existing ones try Chrome DevTools Protocol directly:

const session = await page.target().createCDPSession();
await session.send('Fetch.enable');
session.on('Fetch.requestPaused', async (event) => {
    const headersModified = event.request.headers;
    headersModified['myHeader'] = 'myValue';
    const parameters = {
        requestId: event.requestId,
        headers: headersArray(headersModified),
    }
    await session.send('Fetch.continueRequest', parameters);
});

function headersArray(headers) {
    const result = [];
    for (const name in headers) {
        if (!Object.is(headers[name], undefined))
            result.push({name, value: headers[name] + ''});
    }
    return result;
}